Can One Use Cohen’s Kappa to Examine Disagreement

Abstract. This research discusses the use of Cohen’s j (kappa), Brennan and Prediger’s jn, and the coefficient of raw agreement for the examination of disagreement. Three scenarios are considered. The first involves all disagreement cells in a rater rater cross-tabulation. The second involves one of the triangles of disagreement cells. The third involves the cells that indicate disagreement by one (ordinal) scale unit. For each of these three scenarios, coefficients of disagreement in the form of j equivalents are derived. The behavior of the coefficients of disagreement in the three situations is studied. The first and the third case pose no particular problem. The j equivalents and the other coefficients can be interpreted as usual. In the second case, problems arise such that the range of disagreement js is restricted because the tables are incomplete. Thus, the standard log-frequency model of rater independence is no longer applicable. When the more general models of quasi-independence are used, negative degrees of freedom can result for smaller tables. Simulation results illustrate the characteristics of the coefficients of disagreement for each of the three scenarios. Empirical data examples are given. Cohen’s (1960) j (kappa) is the most popular coefficient of rater agreement. j indicates the degree to which two raters agree beyond chance. It is a summary measure of agreement in a rater rater cross-classification. Re-searchers often ask additional questions concernin