Item Response Theory-based approaches for computing cut scores from an Angoff -based standard setting study

Even when the scoring of the examination is based on Item Response Theory, standard setting methods seldom use this information directly when determining the cut score for the examination from an Angoff-based standard setting study. Typically, the item performance estimates provided by the judges are simply added, regardless of the item parameter estimates for these items. Once these individual judges\u27 totals are determined they are averaged across judges to determine the recommend cut score for the examination. When IRT scoring is used, this total score on the test can be converted to an IRT-based theta ability estimate by using the test characteristic curve. The purpose of this study was to investigate two approaches and its variations to determining the cut score from an Angoff-based standard setting study when the test questions have been calibrated using IRT methods. The first approach (called the Judge Theta Approach) converted the judges\u27 item performance estimates (i.e., probability of correct response) to a theta estimate based on a 3PL IRT model. These theta estimates were then combined in various ways to calculate the cut score. The second approach (called the Simulated Item Vector Approach) used 0/1 values for each item that was determined from the judges\u27 item performance estimates. Again, these 0/1 values were then used a variety of ways to generate a 3PL IRT-based cut score. Both variations of Judge Theta and variations 2 and 3 of Simulated Item Vector approach produced promising results