Assessing Agreement on Classification Tasks: The Kappa Statistic

Introduction Computational linguistic and cognitive science work on discourse and dialogue relies on subjective judgments. For instance, much current research on discourse phenomena distinguishes between behaviours which tend to occur at or around discourse segment boundaries and those which do not (PL93; KID92; LH90; Cah92). Although in some cases discourse segments are defined automatically (e.g., Rodrigues and Lopes' (RL92) definition based on temporal relationships), more usually discourse segments are defined subjectively based on the intentional structure of the discourse, and then other phenomena are related to them. At one time, it was considered sufficient when working with such judgments to show examples based on the authors' interpretation (paradigmatically, (GS86), but also countless others). Research was judged according to whether or not the reader found the explanation plausible. Now, researchers are beginning to require evidence that people besides the authors