Is inter-rater reliability of Global Trigger Tool results altered when members of the review team are replaced?

Objective - To evaluate the inter-rater reliability of results from Global Trigger Tool (GTT) reviews when one of the three reviewers remains consistent, while one or two reviewers rotate. Design - Comparison of results from retrospective record review performed as a cross-sectional study with three review teams each consisting of two non-physicians and one physician; Team I (three consistent reviewers), Team II (one of the two non-physician reviewers or/and the physician from Team I are replaced for different review periods) and Team III (three consistent reviewers different from reviewers in Team I and Team II). Setting - Medium-sized hospital trust in Northern Norway. Participants - A total of 120 records were selected as biweekly samples of 10 from discharge lists between 1 July and 31 December 2010 for a 3-fold review. Intervention - Replacement of review team members was tested to assess impact on inter-rater reliability and adverse events measurment. Main Outcome Measure(s) - Inter-rater reliability assessed with the Cohen kappa coefficient between different teams regarding the presence and severity level of adverse events. Results - Substantial inter-rater reliability regarding the presence and severity level of adverse events was obtained between Teams I and II, while moderate inter-rater reliability was obtained between Teams I and III. Conclusions - Replacement of reviewers did not influence the results provided that one of the non-physician reviewers remains consistent. The experience of the consistent reviewer can result in continued consistency in interpretation with the new reviewer through discussion of events. These findings could encourage more hospital to rotate reviewers in order to optimize resources when using the GTT