Is that measure really one-dimensional? Nuisance parameters can mask severe model misspecification when assessing factorial validity

Factorial validity is often assessed using confirmatory factor analysis. Model fit is commonly evaluated using the cutoff values for the fit indices proposed by Hu and Bentler (1999). There is a body of research showing that those cutoff values cannot be generalized. Model fit does not only depend on the severity of misspecification, but also on nuisance parameters, which are independent of the misspecification. Using a simulation study, we demonstrate their influence on measures of model fit. We specified a severe misspecification, omitting a second factor, which signifies factorial invalidity. Measures of model fit showed only small misfit because nuisance parameters, magnitude of factor loadings and a balanced/imbalanced number of indicators per factor, also influenced the degree of misfit. Drawing from our results, we discuss challenges in the assessment of factorial validity