Analysis of unintended events in hospitals: inter-rater reliability of constructing causal trees and classifying root causes

Background. Root cause analysis is a method to examine causes of unintended events. PRISMA (Prevention and Recovery Information System for Monitoring and Analysis) is a root cause analysis tool. With PRISMA, events are described in causal trees and root causes are subsequently classified with the Eindhoven Classification Model (ECM). It is important that root cause analysis tools are reliable, because they form the basis for patient safety interventions. Objectives. Determining the inter-rater reliability of descriptions, number and classifications of root causes. Design. Totally, 300 unintended event reports were sampled from a database of 2028 events in 30 hospital units. The reports were previously analysed using PRISMA by experienced analysts and were re-analysed to compare descriptions and number of root causes (n ¼ 150) and to determine the inter-rater reliability of classifications (n ¼ 150). Main outcome measures. Percentage agreement and Cohen’s kappa (k). Results. Agreement between descriptions of root causes was satisfactory: 54% agreement, 17% partial agreement and 29% no agreement. Inter-rater reliability of number of root causes was moderate (k ¼ 0.46). Inter-rater reliability of classifying root causes with the ECM was substantial from highest category level (k ¼ 0.71) to lowest subcategory level (k ¼ 0.63). Most discrepancies occurred in classifying external causes. Conclusions. Results indicate that causal tree analysis with PRISMA is reliable. Analysts formulated similar root causes and agreed considerably on classifications, but showed variation in number of root causes. More training on disclosure of all relevant root causes is recommended as well as adjustment of the model by combining all external causes into one category.