Evolution of dominance under frequency-dependent intraspecific competition

A population-genetic analysis is performed of a two-locus two-allele model, in which the primary locus has a major effect on a quantitative trait that is under frequency-dependent disruptive selection caused by intraspecific competition for a continuum of resources. The modifier locus determines the degree of dominance at the trait level. We establish the conditions when a modifier allele can invade and when it becomes fixed if sufficiently frequent. In general, these are not equivalent because an unstable internal equilibrium may exist and the condition for successful invasion of the modifier is more restrictive than that for eventual fixation from already high frequency. However, successful invasion implies global fixation, i.e., fixation from any initial condition. Modifiers of large effect can become fixed, and also invade, in a wider parameter range than modifiers of small effect. We also study modifiers with a direct, frequency-independent deleterious fitness effect. We show that they can invade if they induce a sufficiently high level of dominance and if disruptive selection on the ecological trait is strong enough. For deleterious modifiers, successful invasion no longer implies global fixation because they can become stuck at an intermediate frequency due to a stable internal equilibrium. Although the conditions for invasion and for fixation if sufficiently frequent are independent of the linkage relation between the two loci, the rate of spread depends strongly on it. The present study provides further support to the view that evolution of dominance may be an efficient mechanism to remove unfit heterozygotes that are maintained by balancing selection. It also demonstrates that an invasion analysis of mutants of very small effect is insufficient to obtain a full understanding of the evolutionary dynamics under frequency-dependent selection