Assessing Severity, Immediacy, and Ideal Setting in Emergency Department Patients: a Pilot Study on Inter-Rater Reliability

Objectives: We conducted a pilot study to assess the inter-rater reliability of ED physician assessments of three simple ED visit attributes – severity, immediacy and ideal setting, with long-term goal of developing a novel ED categorization system. Methods: Using 2010 National Hospital Ambulatory Care Survey (NHAMCS) data, we randomly selected 300 ED patient records for review by six board-certified emergency medicine physicians. Each physician reviewed 100 cases where each visit was assessed by two raters on visit severity and immediacy using a 7-point Likert scale, and which of six settings was “ideal” for the visit. De-identified patient demographics, complaints, time of day, day of week, triage assessment, ED care delivered (i.e. testing and medications), disposition and final diagnosis were used for assessments. Respondents classified the certainty of each assessment into “Very Certain” v. “Less Certain”. Weighted kappa and interclass correlation coefficients were used to assess inter-rater agreement. Results: The included study population assessed represented a wide range of illnesses, demographics, diagnoses, ED care and disposition: age averaged 38 years, 56% were female, 34% were non-white, and 13% admitted. For severity, immediacy, and ideal setting there was fair to moderate agreement between the two raters with a weighted kappa of 0.33 (95% confidence interval [CI] 0.27-0.40), 0.30 (95% CI .23-.36), and 0.28 (95% CI 0.21-.34), respectively. When both raters felt “Very Certain” about their assessments weighted kappas were higher, but still only demonstrated fair to moderate agreement for severity 0.42 (95% CI 0.34-.51) and immediacy 0.35 (0.27-0.44). Ideal setting was not stratified by rater certainty. Conclusions: In this pilot study, rater agreement was fair to moderate for severity immediacy, and “ideal” treatment settings for a random sample of ED patients, and only somewhat improved when the raters felt “Very Certain” about assessments. Therefore, even when assessed by expert reviewers, reliably rating ED visit characteristics is a challenge, demonstrating a fundamental validity issue in categorization systems for ED visits