Estimating the variance of male fecundity from genotypes of progeny arrays: evaluation of the Bayesian forward approach

1. Characterizing fine scale mating patterns in plant populations makes it possible to investigate genetic drift, gene flow and selection gradients at a contemporary time scale.Molecular markers are valuable tools for this type of analysis, and numerous statistical methods have been developed to make the best of the information they can provide. In particular, we recently proposed a Bayesian approach based on a paternity analysis wherein we estimate jointly the variance in male fecundity and the pollen dispersal kernel. 2. Here, we use simulated data sets to investigate the accuracy of the Bayesian approach compared to (i) classical maximumlikelihood approaches (e.g. Neighbourhood model) that ignore variance in male fecundity or explain it through a few covariates and (ii) indirect methods (KinDist and Two-Gener) that integrate the variance in fecundity in an ‘effective population density’. 3. The Bayesian estimates correctly considered the over-dispersion resulting from the variance infecundity, resulting in wider but more accurate confidence intervals, in particular in high-densitypopulations. The maximum likelihood methods resulted in confidence intervals with low coverageprobabilities and in widespread false-positive tests when testing the effect of covariates on malefecundity. 4. Estimated individual fecundities and estimated empirical variance in fecundity were robust tothe distribution assumed for the individual random fecundities (log-normal or Gamma). In contrast,the theoretical variance estimate critically depended on the assumed distribution. 5. The indirect methods provided much more variable estimators, as expected because they use lessinformation about pollen sources and consider the molecular information only through genetic structure indices. 6. Disentangling the fecundity from the spatial effects in paternity analyses is necessary when studying selection in natura and or when addressing the effects of spatial distribution on effective gene flow. The Bayesian approach studied here successfully accounts for the variance in fecundity when a large fraction of it is not explained by the studied covariates. The Mixed Effect Mating Model computer program introduced here is devoted to its implementation