Variance heterogeneity in psychological research: A Monte-Carlo study of the consequences for meta-analysis

Variance heterogeneity is common in psychological research. Surveys of psychological research show that variance ratios (VRs) in two-group studies average around 2.5, with a substantial minority of studies having much higher VRs. Research has established that variance heterogeneity disturbs Type I error rates of parametric tests in primary research. Fixed-effects meta-analysis is a common statistical method in psychology for synthesizing primary research, and plays an important role in cumulative science and evidence-based practice. Little is known about the consequences of variance heterogeneity for meta-analytic estimates. The present research reports a Monte Carlo study in which the results of k = 8 or 20 primary studies were generated from each of the distributions N(100, 15) and N(106, 15), for δ = 0.40 (effect size). Variance heterogeneity was created by contaminating the second distribution with elements from a N(106, 45) distribution in proportions ranging from 0.00 to 0.25, to achieve VRs ranging from 1.0 to 3.0. Each simulated fixed-effects meta-analysis (5000 replications) yielded the following estimates: Hedges’ g, CI95% coverage, and I2. In the baseline (VR = 1.0) simulation, g = 0.40 and CI95% coverage = 0.950. In general, larger VRs at the primary-study level were associated with smaller Hedges’ gs and poorer CI95% coverage at the meta-analytic level. For example, at VR = 2.6, g = 0.30 and CI95% coverage = 0.801. In other words, a meta-analysis of studies that simulated the average VR in psychological research substantially underestimated the true effect and inflated the Type I error rate. Study-level variance heterogeneity also inflated estimates of between-study variance (I2), which has implications for meta-regression modeling. This study demonstrates that widely used meta-analytic methods do not produce accurate parameter estimates in the presence of study-level variance heterogeneity