Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation

The practice of using covariates in experimental designs has become controversial. Traditionally touted by statisticians as a useful method to soak up noise in a dependent variable and boost power, the practice recently has been recast in a negative light because of Type I error inflation. But in order to make informed decisions about research practices like this one, researchers need to know more about the actual size of the benefits and costs of these practices. In a series of simulations, we compared the Type I error rates and power of two analytic practices that researchers might use when confronted with an unanticipated, independent covariate. In the baseline practice, a researcher only analyzes the effect of the manipulation on the dependent variable; in the flexible-covariate practice, she analyzes both the effect of the manipulation on the dependent variable and the effect adjusting for the unanticipated covariate. We show that the flexible-covariate (vs. baseline) practice inflates Type I error by a small amount, and that it boosts power substantially under certain circumstances. The flexible-covariate practice tends to be most beneficial when the covariate is strongly correlated with the dependent variable in the population, and when the experimental design would have been only moderately powered (40%–60%) without including the covariate in the analysis. We offer concrete recommendations for when and how to use independent covariates in experimental designs, and contextualize our findings within the movement toward quantifying trade-offs in choosing among research practices and optimizing the choice of practice within a given research context