Assessing and testing interrater agreement on a single target using multiitem rating scales’, Applied psychological measurement 25: 89–99

An index for assessing interrater agreement with respect to a single target using a multi-item rating scale is developed. The variance of rater mean scale scores is used as the numerator of the agreement index. Four variants of a disattenuated agreement index that vary in the random response term used as the denominator are examined. A variant of r ′ wg(J) using perfectly collinear, uniformly distributed responses as the baseline is recommended as a useful index of disattenuated agreement. A test of the equality of variances is recommended for deleting items with a level of interrater agreement significantly different from zero but lower than that of the remaining items. Index terms: aggregation, cross-level analysis, interrater agreement. A situation requiring ratings of a single target on multiple items is common in research. This situation can be found in a variety of psychological research settings, such as those involving students ’ perceptions of the characteristics of a school, employees ’ perceptions of organizational climate, and neighborhood group members ’ perceptions of conditions in their community. Before characterizing the mean (across raters) of the scale scores as representative of the perceptions of the entire group, it is desirable to assess the degree of interrater agreement. To accomplish this, James, Demaree, & Wolfe (1984) proposed a disattenuated agreement index that compares the observed variance in the raters ’ responses to the variance that would be expected if the ratings were characterized by uniformly distributed random responses. Essentially, they applied a Spearman-Brown correction, rwg(J)