Effects of Rater Training on the Assessment of L2 English Oral Proficiency

The main objective of this study was to examine whether a Rater Identity Development (RID)program would increase interrater reliability and improve calibration of scores againstbenchmarks in the assessment of second/foreign language English oral proficiency. Elevenprimary school teachers-as-raters participated. A pretest–intervention/RID–posttest designwas employed and data included 220 assessments of student performances. Two types ofrater-reliability analyses were conducted: first, estimates of the intraclass correlationcoefficient two-way random effects model, in order to indicate the extent to which raterswere consistent in their rankings, and second, a many-facet Rasch measurement analysis,extended through FACETS®, to explore variation regarding systematic differences of raterseverity/leniency. Results showed improvement in terms of consistency, presumably as aresult of training; simultaneously, the differences in severity became greater. Results suggestthat future rater training may draw on central components of RID, such as core concepts inlanguage assessment, individual feedback, and social moderation work.Equity in Assessment (EquA) - Likvärdig bedömnin