Theory of estimation and testing of effect sizes: Use in meta analysis

ABSTRACT. Approximations to the distribution of a common form of effect size are presented. Single sample tests, confidence interval formulation, tests of homogeneity and pooling procedures are based on these approximations. Caveats are presented concerning statistical procedures as applied to sample effect sizes commonly used in meta-analysis. When policy decisions are based on research, such decisions are generally not based on one research study, but on the consensus of many, nor on statistical significance alone, but on evaluation of practical significance (i.e., cost-effectiveness) as well. In recent years there has been growing emphasis on systematic syntheses of the results of research studies, meta-analysis (Glass, McGaw, & Smith, 1981), based on the use of quantitative measures indicating practical as well as statistical significance: effect sizes. The application of such methods has, in many ways, preceded the development of sound mathematical theory necessary to implement its application. Consequently, there is much controversy and doubt about the validity of results based on meta-analysis. The focus here is on the most frequently used sample effect size: d = (xE-xc)/S, where xE is the mean response of nE subjects in an experimental group, xc that of nc subjects in a separate control group, and S2 is a sample variance. Generally, it is assumed that xEi~%(iiE9o£)9 / = 1,2,..., * £, x C / ~9l ( /x c, a<?) , / = 1,2,...,w c