Trip Through the Misleading Characteristics of Inter-Observer Agreement Scores in Interval Recording

The purpose of this project was to assess the degree to which the most popular method of calculating reliability of interval data, the I-I method, served the author's scientific purposes. Data were collected and analyses performed to assess the adequacy of I-I reliability scores in serving three functions: (1) as an index of how precise, clear, objective and complete the definition is; (2) as an index of how competently the observer is recording; and (3) as an index of the believability of the experimental effect reported. Findings from the study indicate gross unreliability of I-I scores as an index of definition adequacy, observer competence or believability of experimental effects. This unreliability is attributed to the fact that I-I scores are highly subject to influence by the rate or duration of the behavior being recorded. Conclusions by the authors stress that, because the I-I reliability scores are clearly inadequate, it is likely that a significant body of applied behavior analysis has seriously misrepresented the relationships between certain environmental factors and certain significant human behaviors. (Author/PC