The limits of unidimensional computerized adaptive tests for polytomous item measures

2013-02-26This dissertation investigated the usefulness of computerized adaptive tests (CATs) based on Samejima’s (1969) unidimensional graded response model (GRM) for polytomous item measures, and had two aims: (1) determine to what extent multidimensionality impacted the accuracy of CAT estimates; and (2) isolate the measure properties impacting the CAT process. The author conducted three studies to address these. The first utilized three measures, each with real data, and evaluated them using standard techniques for measure reduction and unidimensional CAT. In addition, the first study related certain measure properties (e.g. number of items) to differences in CAT and short form performance. The second study, a series of simulations, evaluated the relationship between unidimensional CAT accuracy and six properties: item pool size, number of response options, item discriminations, item difficulty (also skew/kurtosis), number of dimensions, and the observed correlation between dimensions. These simulations suggested dimensionality was most important for accuracy, consistent with past literature. Discriminations better explained mean number of items selected by the CAT, while more than 12 items were recommended for unidimensional CAT to perform better than the short form methods. The final study compared unidimensional results from Study 1 to dimensions from more valid multidimensional models. Results indicated unidimensional CATs handled minor multidimensionality (e.g. two highly correlated dimensions) well. More was too detrimental to the CAT process, though good discriminators may have provided some protection when there was evidence of a strong general dimension. The implications of this for multidimensional adaptive tests utilizing presumably unidimesnional subscales are discussed