A Monte Carlo Study of Fit Indices in Hierarchical Linear Models

In educational research, students often exist in a multilevel social setting that can be identified by students within classrooms, classrooms nested in schools, schools nested in school districts, school districts nested in school counties, and school counties nested in states. These are considered hierarchical, nested, or multilevel because students are within the same community and share similar experiences which have the potential to influence an outcome. Because students within the same classrooms have similar characteristics, conclusions made on these students cannot be independent. To adapt to the hierarchical, multilevel, or nested data structure, multilevel analysis techniques such as hierarchical linear modeling (HLM) can be used to analyze the data. One purpose of HLM is to specify a model that includes appropriate random effects (Guo, 2005). One method which should be considered for inclusion or exclusion of random effects and to evaluate the goodness of fit of the final model to the data is the comparison of models with different specifications of random effects based on Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC) , or Deviance Information Criterion (DIC) which corrects for bias induced by sample size and number of random effects. AIC, BIC, and DIC are information criteria that measure the statistical fit of a model. There has not been any research conducted in the multilevel literature about the impact of sample size and information criteria. This Monte Carlo Monte Carlo simulation compared the influence of sample size on the ability to select the best model in two-level hierarchical models using the information criteria Akaike Information Criterion, Bayesian Information Criterion, and Deviance Information Criterion. Results of this investigation showed that all three information criteria had very low or nonexistent success in choosing the best hierarchical linear model