Variation in heteroploid reproduction and gene flow across a polyploid complex: One size does not fit all

Abstract Whole‐genome duplication is considered an important speciation mechanism in plants. However, its effect on reproductive isolation between higher cytotypes is not well understood. We used backcrosses between different ploidy levels and surveys of mixed‐ploidy contact zones to determine how reproductive barriers differed with cytotype across a polyploid complex. We backcrossed F1 hybrids derived from 2X‐4X and 4X‐6X crosses in the Campanula rotundifolia autopolyploid complex, measured backcross fitness, and estimated backcross DNA cytotype. We then sampled four natural mixed‐ploidy contact zones (two 2X‐4X and two 4X‐6X), estimated ploidy, and genotyped individuals across each contact zone. Reproductive success and capacity for gene flow was markedly lower for 2X‐4X than 4X‐6X hybrids. In fact, 3X hybrids could not backcross; all 2X‐4X backcross progeny resulted from neotetraploid F1 hybrids. Further, no 3X individuals were found in 2X‐4X contact zones, and 2X and 4X individuals were genetically distinct. By contrast, backcrosses of 5X hybrids were relatively successful, particularly when crossed to 6X individuals. In 4X‐6X contact zones, 5X individuals and aneuploids were common and all cytotypes were largely genetically similar and spatially intermixed. Taken together, these results provide strong evidence that reproduction is low between 2X and 4X cytotypes, primarily occurring via unreduced gamete production, but that reproduction and gene flow are ongoing between 4X and 6X cytotypes. Further, it suggests whole‐genome duplication can result in speciation between diploids and polyploids, but is less likely to create reproductive barriers between different polyploid cytotypes, resulting in two fundamentally different potentials for speciation across polyploid complexes