A Comparative Simulation of Type I Error and Power of Four Tests of Homogeneity of Effects For Random- and Fixed-Effects Models of Meta-Analysis

In a Monte Carlo analysis of meta-analytic data, Type I and Type II error rates were compared for four homogeneity tests. The study controlled for violations of normality and homogeneity of variance. This study was modeled after Harwell (1997) and Kromrey and Hogarty\u27s (1998) experimental design. Specifically, it entailed a 2x3x3x3x3x3x2 factorial design. The study also controlled for between-studies variance, as suggested by Hedges and Vevea\u27s (1998) study. As with similar studies, this randomized factorial design was comprised of 5000 iterations for each of the following 7 independent variables: (1) number of studies within the meta-analysis (10 and 30); (2) primary study sample size (10, 40, 200); (3) score distribution skewness and kurtosis (0/0; 1/3; 2/6);(4) equal or random (around typical sample sizes, 1:1; 4:6; and 6:4) within-group sample sizes;(5) equal or unequal group variances (1:1; 2:1; and 4:1);(6)between-studies variance, tau-squared(0, .33, and 1); and (7)between-class effect size differences, delta(0 and .8). The study incorporated 1,458 experimental conditions. Simulated data from each sample were analyzed using each of four significance test statistics including: a)the fixed-effects Q test of homogeneity; b)the random-effects modification of the Q test; c) the conditionally-random procedure; and d)permuted Qbetween. The results of this dissertation will inform researchers regarding the relative effectiveness of these statistical approaches, based on Type I and Type II error rates. This dissertation extends previous investigations of the Q test of homogneity. Specifically, permuted Q provided the greatest frequency of effectiveness across extreme conditions of increasing heterogeneity of effects, unequal group variances and nonnormality. Small numbers of studies and increasing heterogeneity of effects presented the greatest challenges to power for all of the tests under investigation