Multidimensional IRT models for Composite Scores

Tests of English Language Proficiency are often designed such that each section of the test measures a single latent ability. For instance an English Proficiency Assessment might consist of sections measuring Speaking, Listening, and Reading ability. However, Overall English Proficiency and composite abilities are naturally multidimensional. This multidimensionality may even include the case where each item measures more than one latent ability. Parameters for these examples can be estimated within a one-dimensional or multidimensional item response modeling framework. This study compares two common strategies for composite scoring in Test of English Language Proficiency to composite scoring strategies derived from multidimensional models. An example is taken from a sample of 12,008 elementary school students who participated in a State English Proficiency Assessment. Factor analytic techniques were performed to verify the dimensional structure of the composite domains for the Oral scale. Item Response Models with separate unidimensional calibration, simultaneous unidmensional calibration, and multidimensional calibration were examined. Model-data fit, information functions, ability estimates, and interpretive capability among the three different types of composite scores were examined.