Observational review and analysis of concussion : a method for conducting a standardized video analysis of concussion in rugby league

Background: Several professional contact and collision sports have recently introduced the use of sideline video review for club medical staff to help identify and manage concussions. As such, reviewing video footage on the sideline has become increasingly relied upon to assist with improving the identification of possible injury. However, as yet, a standardized method for reviewing such video footage in rugby league has not been published. The aim of this study is to evaluate whether independent raters reliably agreed on the injury characterization when using a standardized observational instrument to record video footage of National Rugby League (NRL) concussions. Methods: Video footage of 25 concussions were randomly selected from a pool of 80 medically diagnosed concussions from the 2013–2014 NRL seasons. Four raters (two naive and two expert) independently viewed video footage of 25 NRL concussions and completed the Observational Review and Analysis of Concussion form for the purpose of this inter-rater reliability study. The inter-rater reliability was calculated using Cohen’s kappa (κ) and intra-class correlation (ICC) statistics. The two naive raters and the two expert raters were compared with one another separately. Results: A considerable number of components for the naive and expert raters had almost perfect agreement (κ or ICC value ≥ 0.9), 9 of 22 (41%) components for naive raters and 21 of 22 (95%) components for expert raters. For the concussion signs, however, the majority of the rating agreement was moderate (κ value 0.6–0.79); both the naive and expert raters had 4 of 6 (67%) concussion signs with moderate agreement. The most difficult concussion sign to achieve agreement on was blank or vacant stare, which had weak (κ value 0.4–0.59) agreement for both naive and expert raters. Conclusions: There appears to be value in expert raters, but less value for naive raters, in using the new Observational Review and Analysis of Concussion (ORAC) Form. The ORAC Form has high inter-rater agreement for most data elements, and it can be used by expert raters evaluating video footage of possible concussion in the NRL