Multilevel modeling of longitudinal growth and change: Substantive effects or regression toward the mean artifacts?

Regression toward the mean artifacts (RTMAs) are ubiquitous phenomena, seducing researchers, policy makers, and practitioners down the path of offering substantive interpretations of statistical artifacts. The use of highly sophisticated statistical tools can mislead otherwise knowledgeable researchers into thinking that this well-known problem is no longer relevant. Here we evaluated multilevel models of growth and change in relation to RMTAs, using simulated data to represent students nested within schools for which there were initial school differences due to selection based on T1 (pretest) achievement scores, but no school differences in achievement growth over four years. In the unconditional multilevel growth modeling approach (i.e., a multilevel repeated measures analysis), there were substantial RTMAs; apparent growth over time was more positive for schools with initially lower T1 school-average achievement and the sizes of these RTMAs varied inversely with the simulated reliability and stability of the measures. In the conditional multilevel covariance approach (i.e., multilevel path models with adjustment for pretest covariates) there were no RTMAs, but some advantages of the growth modeling approach were lost. In a hybrid approach (growth models of residualized change scores) in which T1 achievement was considered as a covariate and growth model components were constructed from T2, T3, and T4 achievement, RTMAs associated with initial achievement differences were eliminated. Because assignment to schools in this example was completely determined by a known T1 (pretest) achievement score, both the conditional and hybrid approaches eliminated RTMAs, but none of the approaches would be generally effective if selection was based on unmeasured variables. The results demonstrate that multilevel growth models provide no protection from RTMAs that can easily be misinterpreted as substantively meaningful results rather than statistical artifacts