Testing Non-nested Multilevel Models

Comparing theories represented by statistical models is central to psychological research. Historically, comparisons between so called “non-nested” models have been error prone in the absence of a null hypothesis test. Recent research by Levy and Hancock and Merkle, You, and Preacher has extended Vuong’s Likelihood Ratio Test of non-nested models to Structural Equation Models (SEMs). A notable omission of recent work is the extension of Vuong’s test to the case of multilevel regression- a common approach for modeling longitudinal or grouped data. This dissertation leverages the similarities between SEMs and multilevel models to extend Vuong’s test to the multilevel framework. The logic of Vuong’s test as it relates to multilevel regression was explored and a SAS macro developed to facilitate the comparison between two models known to be non-nested a priori. The ability of Vuong’s test to select the true or “best” model was compared to that of information criteria in three simulation studies reflecting scenarios in which non-nestedness is commonly encountered in multilevel regression: non-nested covariate sets, level 1 residual covariance structures, and functional forms. Selection rates of the incorrect models were also examined. Vuong’s test showed almost no incorrect model selection across all scenarios, although its power to select the correct model was generally modest. Model comparisons among information criteria tended to be more sensitive than Vuong’s test but also selected the incorrect model more often. Finally, Vuong’s test was applied to three real data sets comparing competing models in the same scenarios as the simulation studies. Implications and recommendations for use are discussed