Monte Carlo simulation comparison of two-stage testing and computerized adaptive testing

Two-stage testing is one variation of adaptive testing, a testing system that adapts the difficulty of the test to the individual\u27s ability level, in an effort to achieve more precise measurement. Two-stage tests have an advantage that they can be administered by paper-and-pencil. Computerized adaptive testing (CAT), another variation of adaptive testing, uses an algorithm to match individual item difficulty to the examinee\u27s ability level, often using an item response theory (IRT). Because of the optimization approach used in CAT, it will be superior in measurement quality to other adaptive approaches. The purpose of this study was to compare an IRT-based two-stage testing to an individualized CAT to ascertain the conditions when two-stage testing might be an acceptably close alternative to CAT in terms of accuracy and efficiency of measurement. Monte Carlo simulation procedures were used to compare three fixed-length CATs (n = 40, 45, 50), and 18 variations of two-stage tests, differing from the length of routing tests (n = 10, 15, 20), distributions of item difficulty parameters in the routing tests (peaked and rectangular), and number of 30-item second-stage measurement tests (6, 7, and 8). As expected, the results indicated that a fixed-length CAT provided superior measurement precision for ability estimation to IRT-based two-stage tests of equivalent length. IRT-based two-stage tests using rectangular distribution of item difficulty in the routing test and an odd number of second-stage tests produced more accurate ability estimates than did other two-stage test configurations studied. Further these ability estimates may be considered as precise as those from the fixed-length CATs. Considering the limitations of CAT implementation and with the practical advantages of two-stage test administration, IRT-based two-stage tests may be a practical and feasible alternative for applications involving a wide range of student ability in school settings