Permutation Tests for Linear Models in Meta-Analysis: An Empirical Investigation of Type I Error Control and Statistical Power Under Non-normality and Variance Heterogeneity. Annual meeting of the Florida Educational Research Association

Meta-analysis is a popular technique in many fields for statistically analyzing and synthesizing results across sets of empirical studies. Meta-analytic techniques provide a variety of models and procedures for pooling effect sizes across studies and for evaluating the effects of potentially moderating variables (Cooper & Hedges, 1994). However, both substantive concerns and statistical concerns about meta-analysis have been raised in the literature. For example, Fern and Monroe (1996) contended that the "interpretation and comparison of effect size across research studies is complicated by differences in substantive problems, theoretical perspectives, research methods, and researchers ' goals " (p. 95). Further, concerns have been raised about the Type I error control and statistical power of meta-analytic tests when key assumptions are violated (Chang, 1993; Harwell, 1997; Kromrey & Hogarty, 1999). The sensitivity of traditional tests in meta-analysis to violations of assumptions is particularly distressing because the tenability of such assumptions (e.g., population normality, variance homogeneity) in the primary studies is often impossible to evaluate unless sufficient details are presented in the reports of primary studies (and such details are frequently not provided; Keselman, et al., 1998). The severity of the concerns that have recently been expressed in the literature suggests tha