S17 Fig -

The effects of ascertaining SNPs polymorphic in archaic humans on real (a) and simulated data (b, c). We focused on 15 f4-statistic classes that are most strongly affected by archaic ascertainment on simulated data (see a list of classes in the legend for panel a). On real data, statistics from these classes were sampled exhaustively and were calculated on AT/GC sites and on archaic-ascertained sites (transitions and transversions), using all sites without missing data at the level of each quadruplet (i.e., using the “allsnps = TRUE” or “useallsnps: YES” setting). Papuans and Australians were excluded from the pool of AMH groups due to their Denisovan ancestry component, which was not simulated; and Africans with substantial non-African ancestry (S1 Table) were also removed to make the distinction between various classes of statistics clearer. Archaic ascertainment was performed either on a group composed of the Altai Neanderthal and Denisovan (panel a, left), or Vindija Neanderthal and Denisovan (panel a, right). A slightly different protocol was used for archaic ascertainment in other parts of this paper since it was performed on a group composed of both Neanderthals and the Denisovan. Graphs illustrating five classes of simulated demographic histories are shown in panel b and scatterplots illustrating the effects of ascertaining SNPs polymorphic in the group composed of one “Neanderthal” and one “Denisovan” individual on genetic data simulated according to those histories are shown in panel c. The same “Neanderthal” and “Denisovan” individuals were used for ascertainment and for calculating f4-statistics, which is a non-optimal (Fig 4A) but inevitable approach in practice. On the graphs (b), the following abbreviations are used: Afr., Africans; nAfr., non-Africans; Den., Denisovan; Neand., Neanderthal. Alternative positions of the out-of-Africa bottleneck simulated at 65, 70, or 75 kya (generation time = 25 years) are marked with red dots. Gene flows from ghost unsampled lineages are shown in green, and those from sampled lineages are shown in blue. Divergence or split times are shown in kya, and effective population sizes are not shown for clarity (see S13 Table). In the first sub-panel (b, Model1) parameters that are the same across all five simulations are shown, and they are omitted from the other sub-panels (b, Models2-5). We focused on 15 f4-statistic classes that are most affected by archaic ascertainment on simulated data (see a list of classes in the legends for each panel). Results for 68 simulated histories are presented in panel c, and by “history” we assume a combination of simulated admixture graph topology, admixture proportions, effective population sizes, population divergence and bottleneck times. The following key simulation parameters are shown in plot titles for Models 1 through 4: model name, proportion of “super-archaic” or Neanderthal-related admixture in ancestral AMH (“Parc”), proportion of Neanderthal or ghost AMH admixture in non-Africans (“PnAfr”), out-of-Africa bottleneck date in kya (“BN”), diploid effective population size during the out-of-Africa bottleneck (“Ne”). The following additional simulation parameters are shown in plot titles for Model 5: proportion of AMH admixture in the Neanderthal lineage (“Pneand”), proportion of Neanderthal admixture in non-Africans (“P1nAfr”), proportion of “ghost AMH” admixture in non-Africans (“P2nAfr”). (PDF)