The shapes of neutral gene genealogies in two species: probabilities of monophyly, paraphyly, and polyphyly in a coalescent model

The genealogies of samples of orthologous regions from multiple species can be classified by their shapes. Using a neutral coalescent model of two species, I give exact probabilities of each of four possible genealogical shapes: reciprocal monophyly, two types of paraphyly, and polyphyly. After the divergence that forms two species, each of which has population size N, polyphyly is the most likely genealogical shape for the lineages of the two species. At ϳ1.300N generations after divergence, paraphyly becomes most likely, and reciprocal monophyly becomes most likely at ϳ1.665N generations. For a given species, the time at which 99% of its loci acquire monophyletic genealogies is ϳ5.298N generations, assuming all loci in its sister species are monophyletic. The probability that all lineages of two species are reciprocally monophyletic given that a sample from the two species has a reciprocally monophyletic genealogy increases rapidly with sample size, as does the probability that the most recent common ancestor (MRCA) for a sample is also the MRCA for all lineages from the two species. The results have potential applications for the testing of evolutionary hypotheses. The genealogy for all copies of a particular genomic region in a particular species can be classified into one of two categories. Either the lineages are monophyletic, that is, they comprise all the extant descendants of their most recent common ancestor (MRCA), or they are not monophyletic. The latter scenario, equivalently, requires that lineages of one or more additional species also descend from this MRCA. Consider all copies of orthologous regions in an ordered pair of species, (A, B). The genealogy of these lineages can be placed into one of four categories: C1, the lineages of each species are separately monophyletic; C2, the lineages of species A are monophyletic, and the lineages of species B are not monophyletic; C3, the lineages of species B are monophyletic, and the lineages of species A are not monophyletic; and C4, Neither the lineages of species A nor the lineages of species B are monophyletic. These scenarios Typically, random pairs of species have reciprocally monophyletic genealogies at nearly all of their orthologous loci. However, the lineages of two closely related species are often not reciprocally monophyletic Probabilities of genealogical shapes for two species have been studied in various population structure models for samples of size 2 I first review models used in the calculations. The main results, namely the probabilities of the four genealogical shapes, are presented in equations THE COALESCENT MODEL: ONE SPECIES I assume a neutral coalescent model to describe the genealogy of the lineages of each of two species (Nordborg 2001). For each species, n lineages, each of which is identified by a distinctive label, are sampled in the present, and the model provides a probability distribution on the set of possible genealogies for these lineages. Going backward in time, the coalescent model has two components: a model that spec