Examining the interrater reliability of the Comprehensive Inventory of Mental Health and Recovery and Rehabilitation Services (CIMHRRS)

This dissertation is one step in the continuing development, evaluation, and validation of the Comprehensive Inventory of Mental Health and Recovery and Rehabilitation Services (CIMHRRS). The CIMHRRS is an instrument to guide comprehensive assessment of programs that provide integrated services to people with serious mental illness (SMI). The CIMHRRS was developed for use in services research and program evaluation. The purpose of the project described in this dissertation was to evaluate three key aspects of its performance in real world application: (1) its practical feasibility, (2) internal consistency and reliability, (3) its ability to distinguish between different programs. The project utilized a combination of principles and methods, associated with psychometric scale development, field methods, and program evaluation. Using a structured site review process, program evaluations were conducted at five SMI service programs that reflect the diversity found in mental health systems. The service programs represent points on a continuum of services for an adult SMI population. Programs varied by location (urban, rural), setting (inpatient, residential, community), security (maximum, medium), service provision, and estimated levels of psychiatric rehabilitation and recovery-oriented services. Investigators assessed program organization, policy and procedures, fidelity to policies and procedures, and outcome. Data collected while on site was used to evaluate the CIMHRRS\u27 practicability and its psychometric properties and determine its capacity to differentiate qualitative aspects of the service programs. Overall, the CIMHRRS demonstrated excellent internal consistency across all subjectively rated items (α=.98) and good to excellent internal consistency within each subjectively rated domains (α=.82 to α=.96). The CIMHRRS demonstrated excellent interrater agreement (97%–100%) and interrater reliability (.99). It demonstrated an ability to differentiate qualitative dimensions of the various programs. The results of this project indicate that the CIMHRRS is a practical, reliable instrument for program evaluation and services research. It is expected to be especially valuable for studying the characteristics of psychiatric rehabilitation, recovery and related approaches to determine their impact on clinical outcome