A Monte Carlo Investigation of Assessment Center Construct Validity Models

There has been some disagreement among three major reviews of the assessment center (AC) construct validity literature (Bowler & Woehr, 2004; Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001) as to the most appropriate analytic model for AC post-exercise dimension rating data. This article reports a Monte Carlo study addressing the following questions with respect to the structure of AC ratings: (a) to what extent does the ‘true ’ model (i.e., the model that generated the data) actually appear to fit the data well? (b) to what extent can a model appear to fit the data well even though it is the wrong model? and (c) is model fit actually a useful empirical criterion for judging which model tested is most likely to hold in the population? Results suggest that the reliance on purely empirical criteria for determining which model best represents the underlying structure of assessment center ratings is potentially problematic: “true ” models may not always appear as the models of best fit and conversely “false ” models sometimes appear to offer better fit than the true models. Moreover, specific parameter estimates differ dramatically depending on the analytic framework in which they are estimated. Implications for the study of assessment center construct validity are discussed