Application of item response models to criterionreferenced test item selection

Of interest in this study was the use of item re-sponse models for obtaining accurate examinee do-main score estimates and for increasing the prob-abilities with which examinees are assigned correct-ly to mastery states with ctiterion-referenced test scores. Specifically, the purpose of this investigation was to compare the one-, two-, and three-parameter logistic test models for estimating domain scores and making mastery/nonmastery decisions. Com-puter simulation methods were used to recover a set of true domain scores with each of the logistic test models under a variety of testing conditions. Also, the percent of times the use of each model led to decisions which were consistent with decisions made with the true domain scores was studied. The one-parameter and three-parameter model resulted in highly comparable results for middle and high abil-ity examinees, while for low ability examinees, the more general model always performed somewhat better. The success of objectives-based programs in education, industry, and the military depends, to a considerable extent, upon the quality of the criterion-referenced tests that are used. Recent technical advances have made it possible to build high quality criterion-referenced tests (CRTS) as well as to carry out evaluations of the tests and the scores derived from them (see, for example, Berk, 1980; Hambleton, 1980; Pop-ham, 1978). Still, several important problems re-main. For example, methods are needed for ob-taining more accurate examinee domain score estimates ’ and for increasing the probabilities with which examinees are assigned correctly to mastery states. Lengthening tests is a common method. However, it is often not practical to lengthen tests, particularly to the lengths they would need to be, to meet desirable levels for the reliability and validity of test scores (Wilcox, 1976). ’f’~11®red/~d~ptl~e testing is a promising method, but at the present time it is somewhat impractical because it normally involves the ad-ministration of test items by a computer termi-nal (Lord, 1980b; Weiss, 1977). The use of Bayesian statistical procedures is another method for improving the accuracy of domain score estimates and mastery decisions (Novick, &amp