Evaluating Methods for Handling Multilevel Selection for the Purpose of Generalizing Cluster Randomized Trials

Over the past decade, the generalizability of randomized experiments, defined as the level of consistency between an estimated treatment effect in a sample and the true treatment effect in a target population, has received increasing attention from the educational research community. Existing methods focus on either: (a) prospectively preventing or (b) retrospectively adjusting away the bias caused by the nonrandom selection of institutions, such as schools, into a study sample. This study explores methods to adjust away the bias caused by both the between-institution and within-institution selection processes. For instance, in educational studies we desire to account for both nonrandom school-level selection and non-random selection of students and/or teachers. I conduct a simulation study to evaluate the bias reducing properties of different methods for estimating and utilizing inverse probability of participation (IPP) weights in this two-level context. The simulation study found that methods that incorporated both student and school IPP weights reduced more bias than methods that only incorporated the school IPP weights. Using data from a cluster randomized trial of a math professional development intervention, this study finds that within participating schools, the participating sample of students were more “advantaged” than the non-participating students, suggesting the need to adjust for within school nonrandom selection. The study estimated and applied IPP weights to reduce bias in the estimated population average treatment effect for the intervention