The effects of instructions and calculation procedures on the accuracy, agreement, and calculation correctness of observers

Behavioral research and therapy often rely on data collected in natural settings by human observers. Research interests have begun to focus on the methodological issues involved when data are collected in this manner. These methodological issues have been categorized into four areas: code complexity: observee reactivity: the measurement of inter-observer agreement: and observer bias. The present study looked at issues related to measuring inter-observer agreement, and attempted to answer the following questions: first, what effect do instructions to observers have on the levels of agreement and observational accuracy they achieve: and second, do instructions observers follow in calculating agreement result in their calculating their own agreement differently than they calculate the agreement levels of other observers. Sixteen undergraduates were trained to use a behavioral code to record from videotapes the classroom behavior of two eight-year-old second grade males, and to calculate inter-observer agreement levels. The subjects were then paired and randomly assigned to one of the two instructions groups. Instructions were to try to reach agreement of .85 or better with one's partner, or to make one's observational recordings as carefully as possible