Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification

This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.Education, Faculty ofEducational and Counselling Psychology, and Special Education (ECPS), Department ofGraduat