International Biometric Society Agreement between two independent groups of raters

Although agreement is often searched between two individual raters, there are situations where agreement is needed between two groups of raters. For example, a group of students may be evaluated against a group of experts or two groups of physicians with different specialty may be challenged in diagnosing patients with the same test (positive/negative). Kappa-like agree-ment indexes are commonly used to quantify agreement between two raters on a nominal or an ordinal scale. They include Cohen’s kappa coefficient (Cohen, 1960), the weighted kappa coefficient (Cohen, 1968) and the intraclass kappa coefficient (Kraemer, 1979). To quantify agreement between two groups of raters, the common practice is simply to determine a con-sensus in each group of raters in order to reduce the problem to the case of two raters. The consensus could be defined by taking the modal category in each group (e.g., van Hoeij and al., 2004) or the median if the scale is ordinal (e.g., Raine and al., 2004). In all cases, however, the question of how to proceed when a consensus can not be reached remains. Moreover, different definitions of consensus may lead to different conclusions (Kraemer and al., 2004) and consensus approaches don’t take the variability in the groups into account. We propose a novel agreement index, not requiring any consensus definition and extending the basic con-cept of Cohen’s kappa coefficient to two groups of raters. The agreement index is defined on a population model and the sampling variance is determined by the Jackknife method