Human Annotation Protocol using a Computer Assisted Method for Validating Electrocardiographic Algorithms to Suppress False Alarms and Reduce Alarm Fatigue

AbstractBackground: Inundation with alarms in the Intensive Care Unit (ICU) creates alarm fatigue forclinicians and contributes to patient safety events including injury and death because true events are missed. Electrocardiographic (ECG) alarms significantly contribute to this problem because nearly 90% are false positives. These false alarms interrupt care, desensitize clinicians to all alarms, and ultimately reduce responsiveness by clinicians. Research directed towards improving ECG alarm algorithms and alarm specificity is being conducted, however, no current standard for human annotation of true versus false events has been established. Objectives: This study was designed to examine annotation methods for analysis of Ventricular Tachycardia (VT) alarms as true versus false and identify challenges for improving these methodologies. The purpose of this study was to: (1) describe a computer-based annotation tool for annotating VT alarms using 24-hour ICU patient files; (2) examine inter-rater reliability between two ICU nurse annotators; and (3) identify specific areas for improving the annotation tool and training preparation of nurse annotators. Results: Annotation of 749 VT alarms by two independent ICU nurse reviewers resulted in agreement of 55% (Kappa 0.216). Reviewer 1 identified 333 (44.5%) of the alarms as false, whereas Reviewer 2 identified 605 (80.8%) as false. Conclusion: In this pilot study, we identified that more training and education of annotators was needed and that the annotation protocol could be improved by reducing the number of annotation categories, single and random presentation of alarms, rather than an entire 24-hour file for one patient, and validation by an ECG expert with education and training developed from disagreements and incorrect annotations