The application of an automated item selection method to real data

A method of automatically selecting items for inclusion in a test that has constraints on item content and statistical properties was applied to real data. Two tests were assembled by test specialists who assemble such tests on a routine basis. Using the same pool of items and the same constraints, the two tests were reassembled auto-matically. Test specialists not involved in the original manual assembly compared the tests constructed manually to the tests constructed automatically. The results indicated that the progress of automated test assembly methods lies in improving item banking systems, classification schemes, and quality control measures, rather than in the development of different algorithms or in the improvement of computer time and cost. Index terms: heuristic algorithms, mathe-matical programming, test assembly, test construction, test design. The process of test construction is time consuming and expensive. Every year public and private testing organizations spend millions of dollars writing and editing items and then assembling these items into test forms for some measurement purpose. Until recently, the process of test construction or test assembly has been virtually unassisted by modern psychometrics. A practical application of item response theory (IRT) suggested by Lord (1980) and Birnbaum (1968) is to use IRT in the test assembly process. But the test assembly process they suggested has little or no practical (as opposed to measurement) advantage over test assembly methods using conventional statistics without additional assistance from computers. In the last 10 years, many test assembly paradigms employing a combination of IRT, computers, and mathematical programming methods or heuristic methods have been proposed (e.g., Ackerman, 1989; Adema, 1988; Baker, Cohen, & Bar