The measurement of interrater agreement

The statistical methods described in the preceding chapter for controlling for error are applicable only when the rates of misclassification are known from external sources or are estimable by applying a well-defined standard classifi-cation procedure to a subsample of the group under study. For some variables of importance, however, no such standard is readily apparent. To assess the extent to which a given characterization of a subject is reliable, it is clear that we must have a number of subjects classified more than once, for example by more than one rater. The degree of agreement among the raters provides no more than an upper bound on the degree of accuracy present in the ratings, however. If agreement among the raters is good, then there is a possibility, but by no means a guarantee, that the ratings do in fact reflect the dimension they are purported to reflect. If their agreement is poor, on the other hand, then the usefulness of the ratings is severely limited, for it is meaningless to ask what is associated with the variable being rated when one cannot even trust those ratings to begin with. In this chapter we consider the measurement of interrater agreement when the ratings are on categorical scales. Section 18.1 is devoted to the case of the same two raters per subject. Section 18.2 considers weighted kappa to incorporate a notion of distance between rating categories. Section 18.3 is devoted to the case of multiple ratings per subject with different sets of raters. Applications to other problems are indicated in Section 18.4. Section 18.5 * relates the results of the preceding sections to the theory presented in Chapter 15 on correlated binary variables