Estimating false alarms and missed events from interobserver agreement: a rationale. Psychol Bull

In the analysis of temporal, sequential, or contingent relations among events within sessions, the observers ' or coders ' false-alarm rate and missed-event rate are more useful than coefficients of interobserver agreement as indices of re-liability. False-alarm and missed-event probabilities can be estimated from com-parisons of two codings, either in the fixed-unit situation or when behavior has been coded in continuous time. Complex problems, such as those that arise when one computes rates in the context of two or more mutually exclusive continuous states, can often be reduced to a simple model, so long as investigators keep in mind the precise meaning of the reliability indices they use. Complex analyses of sequential data are be-coming increasingly important to investiga-tors of ue. ' avior in its natural contexts, in-cluding the interpersonal behavior of dyads or groups. The data in such studies often con-sist of series of observed events (Kaye, 1978). These may be coded continuously as they oc-cur, or in slow motion, or by a stop-frame method, or in unit time intervals checked as containing or not containing an event of a given kind, or even from a transcript in which an utterance, answer, or trial takes the place of unit time. So long as one is only concerned with a sum-mary measure from each subject or session, such as the frequency or rate with which a par-ticular category of behavior occurs, reliability is a matter of how well the measure discrimin-ates among the subjects (correlation over ob-servers, over sessions, etc.), and validity is a matter of how well the measure generalizes to the subjects ' behavior in other situations. This article is concerned with cases in which that is not sufficient. When data are to be microanalyzed—that is, whenever one is con-I am grateful to Donald Fiske, William Kruskal, Lalitha Sanathanan, and Jeanne Altmann for helpful comments on earlier drafts of this article. Requests for reprints should be sent to Kennet