Reviewing evidence on complex social interventions: appraising implementation in systematic reviews of the health effects of organisational-level workplace interventions

Objective: We appraised the reporting of intervention implementation in studies included in systematic reviews of organisational-level workplace interventions. Implementation is taken to include such factors as intervention setting, resources, planning, collaborations, delivery and macro-level socio-economic contexts. Understanding how implementation affects intervention outcomes may help prevent erroneous conclusions and misleading assumptions about generalisability, but implementation must be adequately reported if it is to be taken into account. Design: Data on implementation was obtained from four systematic reviews of complex interventions in workplace settings. Implementation was appraised using a specially-developed checklist, and by means of an unstructured reading of the text. Results: We identified and appraised 103 studies evaluating 4 types of organisational-level workplace intervention (employee participation, changing job tasks, shift changes and compressed working weeks). Many studies referred to implementation but reporting was generally poor and anecdotal in form. This poor quality of reporting did not vary greatly by type or date of publication. A minority of studies described how implementation may have influenced outcomes. These descriptions were more usefully explored through an unstructured reading of the text, rather than by means of the checklist. Conclusions: Evaluations of complex interventions should include more detailed reporting of implementation and consider how to measure quality of implementation. Our checklist helped us explore the poor reporting of implementation in a more systematic fashion. In terms of interpreting study findings and their transferability, the more qualitative appraisals however appeared to offer greater potential for exploring how implementation may influence the findings of specific evaluations. Implementation appraisal techniques for systematic reviews of complex interventions require further development and testing