Assessing The Dimensionality of Polytomous Item Exams

Unidimensionality is a main assumption for Item Response Theory (IRT) models that are applied to statewide standardized tests for purposes such as scoring, equating, scaling, and DIF analyses. Thus, it is important that the unidimensionality evaluation of tests be efficiently and effectively carried out on a routine basis. While the vast majority of statewide standardized tests contain both dichotomous and polytomous items, much of the work in dimensionality assessment has focused on the case of dichotomous item exams. The proposed method in this dissertation, Poly-NEWDIM (PND), demonstrates closer to nominal Type I error control and better power than two currently existing methods, the original Poly-DIMTEST (Li & Stout, 1995; Nandakumar et al., 1998), and the polytomous extension of DIMTEST without AT2 (Froelich, 2000). The first step in using Poly-NEWDIM is to decide on the AT and PT subtests. The proposed two new AT-selection procedures in this dissertation, HCPO-AF and HCPND, perform better than the previous AT selection method (HCDE; Froelich & Habing, 2008) in terms of power. Moreover, HCPND shows more robustness than HCPO-AF to sample size, correlation, and structure complexity. The results also indicate that 70% of data for AT selection is appropriate for all three kinds of tests (dichotomous only, polytomous only, and mixed) with large sample sizes and dichotomous tests with small sample sizes, while 50% is good for polytomous tests and mixed tests with small sample sizes. All of the dimensionality assessments used in this dissertation are theoretically founded on the conditional covariance theory. It is necessary and important to investigate the influence on the effectiveness of conditional covariance theory when the assumption of the compensatory model is violated. The results of a simulation study using the variable compensation model showed that conditional covariance theory performed worse when the generating model is further from the theory-assumed compensatory model, but not as badly if the reliability was well controlled for the simulated test