The Effects of Rater Training on the Relationship between Item Observability and Rater Agreement

This study was an extension of a study conducted by Roch, Paquin, and Littlejohn (2009). They investigated the relationship between rater agreement and the observability of items on a rating form. The current study found similar results in that, as items became less observable, interrater agreement increased. The purpose of this study was to introduce frame of reference training as an extension to the Roch et al. study in order to reverse their findings. In other words, trained raters would be less likely to default to a general impression on less observable items and thus would demonstrate higher rater agreement on more observable items than untrained raters. The results, based on 66 raters, replicated the findings of the Roch et al. study. The frame of reference training appeared to have no impact on the results. Results are discussed