Comparative analysis of experimental data

1.We consider the problem of how to analyse data from experiments conducted on multiple species. This seems to have been largely overlooked in the literature, and we highlight that the use of species as experimental units creates issues for both the design and analysis of experiments. 2.We distinguish fully randomized experiments in which all treatments are applied to all species from those experiments in which the factor of interest varies at the species level, i.e. treatments are not randomly allocated to species. In this latter case, the distribution of the experimental factor across species may be random, phylogenetically structured, or species may be chosen in order to phylogenetically balance the sample (e.g. through sister‐species comparisons). 3.We show using simulations that the design of the experiment in terms of the phylogenetic distribution of the treatment can affect power and Type I error, and that commonly used approaches (Linear Mixed Models and ANOVAs) may have poor statistical properties (high Type I errors) when both the predictors and response data show strong phylogenetic signal. 4.We highlight that the true phylogenetic generalized least squares model yield has good statistical properties but show that in some cases the true variance structure may be difficult to identify empirically. Moreover many current comparative tools do not allow such analyses to be easily applied, and we highlight some of those that do