Validity Generalization and Transportability: An Investigation of Distributional Assumptions of Random-Effects Meta-Analytic Methods

Validity generalization work over the past 25 years has called into question the veracity of the assumption that validity is situationally specific. Recent theoretical and methodological work has suggested that validity coefficients may be transportable even if true validity is not a constant. Most transportability work is based on the assumption that the distribution of rho ( ρi ) is normal, yet, no empirical evidence exists to support this assumption. The present study used a competing model approach in which a new procedure for assessing transportability was compared with two more commonly used methods. Empirical Bayes estimation (Brannick, 2001; Brannick & Hall, 2003) was evaluated alongside both the Schmidt-Hunter multiplicative model (Hunter & Schmidt, 1990) and a corrected Hedges-Vevea (see Hall & Brannick, 2002; Hedges & Vevea, 1998) model. The purpose of the present study was two-fold. The first part of the study compared the accuracy of estimates of the mean, standard deviation, and the lower bound of 90 and 99 percent credibility intervals computed from the three different methods across 32 simulated conditions. The mean, variance, and shape of the distribution varied across the simulated conditions. The second part of the study involved comparing results of analyses of the three methods based on previously published validity coefficients. The second part of the study was used to show whether choice of method for determining whether transportability is warranted matters in practice. Results of the simulation analyses suggest that the Schmidt-Hunter method is superior to the other methods even when the distribution of true validity parameters violates the assumption of normality. Results of analyses conducted on real data show trends consistent with those evident in the analyses of the simulated data. Conclusions regarding transportability, however, did not change as a function of method used for any of the real data sets. Limitations of the present study as well as recommendations for practice and future research are provided