Evaluation in a policy environment : approaches to the evaluation of complex health policy pilots in the UK from 1994 to 2004

Pilots have become a common feature of public policy-making. Each has been subjected to comprehensive centrally commissioned national evaluation. Evaluation is therefore well placed to make a significant contribution to public policy, but is it up to the challenge? This study was necessary for two reasons. First, there is a lack of consensus in the literature concerning the purpose of policy evaluation and the optimal ways both to generate and use knowledge within a policy environment. Second, the empirical evidence concerning evaluation’s ability to thrive in a policy environment has been mixed and comes largely from a number of single evaluation case studies, which are limited by their attention to a single research design. Thus, more evaluation of policy evaluation is needed in order to provide a sound base for theory development and improved practice. A comparative collective case study was undertaken of the evaluations of four UK health policy pilots from 1994 and 2004 to ascertain whether the dissensus in the literature was reflected in evaluation practice and to consider how the insights gained might contribute to the medium-term future of health policy evaluation. This study’s contribution starts from its conclusion that there is much to be gained from a more cumulative approach to policy evaluation scholarship; it proposes a realist, integrative, ideal-type theoretical framework for health policy evaluation, which brings together an understanding of how evaluation can thrive in a policy environment with an understanding of how the evaluation of complex innovations can be undertaken effectively.