Setting standards: a comparison of two Rasch-Model methods

Master of EducationStandard setting is an important issue in education. This study has applied two Rasch model techniques, Angoffs sum of probability and Andrich's pairwise comparison, to set cutoff scores for a Tasmanian Year 9 Link Test at year 3, year 5 and year 8 level. The main focus of the study was to examine methods of setting cutoff scores in the context of the National Profiles. The study also aimed at examining the relative difficulty of two parts of the Link Test on Mathematics for Year 9 students in Tasmania. Standard setting issues were analysed in the light of criterion referenced assessment in terms of their purposes and implications in education and measurement. Review of previous standard setting methods was also done. The two methods applied in the study were Angoff's sum of probabilities and Andrich's pairwise comparison. Both are popular in the educational context for establishing cutoff scores and examining the difficulty levels of tests. The findings produced by the sum of probability method have shown that cutoff score at each level of part 1 was higher than that of part 2, so that part 2 of the Link Test was more difficult than part 1. It was consistent with the item estimates produced by the pairwise comparison method. The correlations were high and significant at 0.05 level. It was consistent to those findings by Andrich and Titmanis (1997). However, the correlations between item difficulty estimates established by the pairwise comparison and empirical data were low from 0.39. The study has shown the possibility of using the two Rasch model techniques, sum of probability and pairwise comparison to establish cutoff score and examine the relative difficulty of the tests. Pairwise comparison method could be used as a mean of checking the validity of the proposed standards. It was shown that further research in this field should be done