In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency

To examine the inter-rater reliability of two risk assessment tools: The Level of Service Inventory-Revised (LSI-R) and the Youth Level of Service/Case Management Inventory (YLS/CMI). Methods: Two identical experiments are reported. For both studies, a random sample of 10 offenders were interviewed and videotaped with each tool (totaling 20 offenders). The tapes were then shown to a random selection of 20 raters (for a total of 40 raters) employed at a state agency. The fully-crossed design allowed each of the raters to rate the each of the cases, resulting in 200 total risk score observations for each tool. Inter-rater reliability analyses were then conducted. Results: The LSI-R demonstrated adequate to fair reliability, with certain domains showing lower reliability. Overall, the LSI-R had an ICC of .65. The YLS/CMI demonstrated higher reliability (ICC of .78). In addition, for the LSI-R study, comparisons were made between staff raters who work in a facility versus those in the community (e.g., probation officers). For the YLS/CMI study, comparisons were made between incarcerated offenders versus probationers. Neither comparison yielded consistent differences. Conclusions: The YLS/CMI is generally reliable. The LSI-R showed less reliability. However, each study showed certain domains with less than ideal reliability