A comparison of two training procedures for maintaining inter-rater reliability

Much behavior modification research has relied on the use of human observers to collect data. However, instrumental errors have been found to be associated with the collection of observational data. The present study compared two types of training procedures in which the consistency of the standard, to which the raters were trained to conform, was varied. Sixteen college undergraduates (Ss) without prior experience in observing classroom behavior were trained in the observation of nursery-school children's behavior. All Ss viewed the same 10-minute videotapes of nursery-school children. Seven 60-minute training sessions were conducted using the O'Leary disruptive behavior code. Four pairs of observers in Group I were trained by one graduate student trainer whose ratings were accepted as the standard. Four pairs of observers in Group II were trained by themselves, thus establishing their own standard. Following training, six videotapes were rated by both groups. Videotapes were divided into four 10-minute blocks to permit the collection of overtly and covertly assessed reliabilities for both within-pair and between-pair combinations