Evaluating the articulation of programme theory in practice as observed in Quality Improvement initiatives

Background: The Action-Effect Method(AEM) was co-developed by NIHR CLAHRC Northwest London (CLAHRC NWL) researchers and QI practitioners, building on Driver Diagrams(DD). This study aimed to determine AEM effectiveness in terms of technical aspects (how diagrams produced in practice compared with theoretical ideals) and social aspects (how engagement with the method related to social benefits). Methods Diagrams were scored on criteria developed on theoretical ideals of programme theory. 65 programme theory diagrams were reviewed (21 published Driver Diagrams (External DDs), 22 CLAHRC NWL Driver Diagrams (Internal DDs), and 21 CLAHRC NWL Action-Effect Diagrams(AEDs)). Social functions were studied through ethnographic observation of frontline QI teams in AEM sessions facilitated by QI experts. Qualitative analysis used inductive and deductive coding. Results ANOVA indicated the AEM significantly improved the quality of programme theory diagrams over Internal and External DDs on an average of 5 criteria from an 8-point assessment. Articulated aims were more likely to be patient-focused and high-level in AEDs than DDs. The cause/effect relationships from intervention to overall aim also tended to be clearer and were more likely than DDs to contain appropriate measure concepts. Using the AEM also served several social functions such as facilitating dialogue among multidisciplinary teams, and encouraging teams to act scientifically and pragmatically about planning and measuring QI interventions. Implications: The Action-Effect Method developed by CLAHRC NWL resulted in improvements over Driver Diagrams in articulating programme theory, which has wide-ranging benefits to quality improvement, including encouraging broad multi-disciplinary buy-in to clear aims and pre-planning a rigorous evaluation strategy