Direct Training to Increase Inter-rater Agreement between an Observer’s and Teachers’ Self-Report Ratings of Treatment Integrity

Measuring an implementer’s treatment integrity, specifically an implementer’s adherence to steps of an intervention, can be done via direct (e.g., observation) or indirect (e.g., self-report) methods of assessment. Direct observation is a widely used and accepted method of data collection in research due to its technical adequacy. However, direct observation is resource intensive, often making it impractical outside of research. Self-report measures of adherence can be less resource intensive and are commonly used in school settings, yet results from previous research indicate that implementers frequently overestimate their adherence when using self-report measures. To address this issue, results from research that build support for teacher self-report as a reliable method of treatment integrity assessment are needed. As such, the objective of the current study was to improve inter-rater agreement (IRA) between teachers’ adherence self-report ratings and ratings provided by an observer. The student investigator (i.e., primary observer) observed instructional practice during baseline. Then, after a brief indirect training on the intervention, the primary observer and teachers rated teacher adherence to an explicit instruction intervention. When it was determined that the teachers’ adherence ratings did not adequately agree with the observer’s, teachers were staggered into a phase in which they received direct training on the intervention steps to assess if a change in IRA occurred. Results indicate that after direct intervention training, IRA between the primary observer and teachers improved