Wrangling with realist synthesis: A case study of evolving practice in the development of a systematic review of school accountability

How do researchers, organizations that support systematic reviews, funders and other stakeholders come to understand what a realist review is and can contribute to important questions in educational policy and practice? This paper presents a case study of the emergence of realist synthesis as a valid practice in relation to a systematic review of school accountability in developing countries. The research team conducting this theory-building realist review (Westhorp, et al, forthcoming 2014, p. 22, fn 7) has methodological expertise in a range of areas related to realist theoretical perspectives and configurative systematic review (Gough, et al., 2012), but none has worked specifically on a systematic realist synthesis (Wong, et al., 2013). The UK Department for International Development, a major funder of systematic reviews in education, is funding the review, and the review is supported by the EPPI Centre, one of the world-leading centres for education-focused systematic reviews. The notion of realist synthesis as an approach to systematic review is relatively novel to both the funder and the support organization. Moreover, realist review is an unknown approach to the highly-experienced practitioners, policy analysts and educational researchers being recruited to the review's advisory group. Nonetheless, funder, sponsor organisation and, potentially, the advisory group are interested in exploring realist approaches to addressing the difficulties confronting reviews of highly-varied sources of information that address the complexities of educational praxis in important policy areas. This instrumental case study (Stake, 1996) focuses on critical incidents (Flanagan, 1954) that arise within the team as well as between the team and those who oversee and will use its work--the funder, sponsor and advisory group. Critical incidents are currently being identified in retrospect, reflected upon through discussion and documented through the consistent use of analytic memos (Maxwell, 2005). The analytic memos will be read as a set and discussed within the research team to identify the pathways through which collective understanding of core concepts and practices has and has not emerged. At the time of this submission, the review protocol has been prepared and approved. Two of the critical incidents being tracked to date include: debate around an initial rough theory and elaboration of embedded, hypothetical middle-range theories that appear to be the most promising loci of theory building; and developing approaches to screening and a coding tool that align with the iterative rhythm of realist review while facilitating challenges to as well as elaboration of initial middle-range theories. The research team is particularly interested in tracing critical incidents that relate working with middle-range theories in the translation of review findings for policy and practice, highlighted by the funder and advisory group members as an overarching concern. This case study intends to add to the emerging body of knowledge around the process of realist review and the development of realist synthesis as a legitimate and institutionalised approach to systematic review in education