Exploring the Behavior of Model Fit Criteria in the Bayesian Approximate Measurement Invariance: A Simulation Study

Measurement invariance (MI) is conducted to ensure that differences found in the results of group comparisons are due to true substantive differences and not methodological artifacts. Previous cross-cultural and cross-national studies with large number of groups showed that the advanced measurement invariance level was rarely held when utilizing the traditional (frequentist) MI approach. The Bayesian approximate measurement invariance (BAMI) was introduced to override the traditional MI strict assumption, because trivial non-invariance in parameters across groups is allowed. Although the concept of the BAMI, which has been utilized since 2013, was incorporated into the context of structural equation modeling, there is still a need for clear-cut criteria of BAMI for group comparison because the Bayesian approach can account for uncertainty when appropriately modeled. Given this, the current study demonstrates the usefulness and flexibility of Bayesian approximate measurement invariance and aims to examine the extent to which employing different research settings would affect the behavior of the BAMI across populations. Particularly, a Monte Carlo study was designed to evaluate the sensitivity of the BAMI model fit criteria to varying prior estimates and simulation conditions. The design factors include the group numbers, percent of groups with the non-invariant item intercepts (balanced and unbalanced), and magnitude and directions of DIF item intercepts. The conditions were chosen based on a systematic literature review of the BAMI applied studies conducted between 2013 and 2017 as well as a review of the BAMI published simulation studies. Crossing all the data generation factors for exact models resulted in a total of 2 simulation conditions, whereas approximate models resulted in a total of 24 simulation conditions. Primarily, the analysis procedure included two modeling approaches. a) exact-zero scalar MI against exact-zero metric MI, and b) Bayesian approximate-zero scalar MI with five level of prior precision variances. The generated data were analyzed using maximum likelihood estimator and Bayes estimator with five different prior variances that were addressed in the literature, .001, .005, .01, .05, and .10. All generated data were fitted to each model. Two BAMI model fit criteria were used (PPP and 95% CI) as well as three model comparisons criteria (Bayes factor, BIC, and DIC). In order to assess the sensitivity of the exact and BAMI model fit criteria, three outcome variables were evaluated as a function of design factors: (a) convergence rates, (b) model fit evaluation for models using maximum likelihood and Bayes estimators, and (c) Type I error and noninvariance detection rates for scalar measurement invariance models under exact MI, approximate MI, and noninvariance conditions. Based on the noninvariance detection rates, a reasonable cutoff of the prior variance of Bayes estimation was assessed. The impact of simulation factors on the performance of exact and BAMI tests was also evaluated. Results highlighted that the choice of the prior size affected the BAMI performance, and suggested three pairs of priors for BAMI, (.001 and .05), (.01 and .05), and (.01 and .10), where the first prior in the pair is a representant of approximate-zero invariance while the second prior in the pair is a representant of the substantial non-invariance. In line with the suitable pair of priors, the results also showed that BAMI performed very well if an appropriate fit criterion was used, (e.g., Bayes factor (BF) with 150 as a cutoff and Deviance information criterion (DIC)). Implications for BAMI researchers and future directions are discussed