A Latent Trait Look at Pretest-Posttest Validation of Criterion-referenced Test Items

Since Cox and Vargas (1966) introduced their pretest-posttest validity index for criterion-referenced test items, a great number of additions and modifica-tions have followed. All are based on the idea of gain scoring; that is, they are computed from the differences between proportions of pretest andposttest item responses. Although the method is simple and generally considered as the prototype of criterion-referenced item analysis, it has many and serious dis-advantages. Some of these go back to the fact that it leads to indices based on a dual test administration- and population-dependent item p values. Others have to do with the global information about the discriminating power that these indices provide, the implicit weighting they suppose, and the meaningless maximization of posttest scores they lead to. Analyzing the pretest-posttest method from a latent trait point of view, it is proposed to replace indices like Cox and Vargas ' Dpp by an evaluation of the item information function for the mastery score. An empirical study was conducted to compare the differences in item selection between both methods. As in any other area of educational and psychological measurement, more attention has been paid to reliability than to validity aspects of criterion-referenced measure ment. Several test parameters have been proposed and compared with their norm-referenced counterparts, assessment methods have been introduced and examined using both real and simulated data, and the criterion-referenced reliability problem seems on its way to a great diversity of solutions (Hambleton & Novick, 1973; Huynh