The potential of systematic reviews in environmental social science : an analysis of its use to evaluate and inform policy

There is common agreement that public policy should be formulated based onknowledge of how it works and makes impact. Both scientific knowledge andevaluations can contribute to environmental decision-making and play animportant role to inform policy development. Over the past two decades,systematic review (SR) methods have been proposed to contribute to thispurpose. SRs are literature reviews, using explicit methods for selecting andanalysing empirical data. This dissertation explores the opportunities andchallenges occurring when introducing SR methods into environmental socialscience. Two main research questions are answered: 1) What are theopportunities and challenges of applying SR methods to investigate policy issuesin general, and environmental policy issues in particular?; 2) How have SRs(process and product) been used to inform decision makers, how can they be usedand how can we understand such use? To this end, I examine how other researchers have used the SR methods (articleI); how project managers, researchers and stakeholders view the usefulness of themethods (article II); I apply a specific SR method to a policy issue (article III);and discuss the relevance of the SR methods in general to political science (articleIV). Multiple methods and materials were used, including an overview ofliterature, a realist review and interviews with project managers, researchers, andstakeholders. The findings can be summarised into four main points. Firstly, SR methods canbe used to evaluate environmental policy, but the challenges in applying themethods to social science policy research should be kept in mind. SR methodsprovide guidelines for how to make a literature review that is rigorous andmethodologically robust, with a number of benefits such as contributing with newempirical results and developing theory, methods and research design. At the same time, a number of challenges arise when they are to be applied to complexissues, heterogeneous research methods and data. Secondly, methodological pluralism should be applied in SR. Given how socialscience research is conducted (with many different methods, both qualitative andquantitative) and which issues are examined (often complex), I advocatemethodological pluralism regarding what should be considered an SR method. Itshould include both qualitative and quantitative methods, without any hierarchybetween them. Thirdly, there is need to nuance the theoretical understanding of how SRs areused and how they can be expected to be used. The findings suggest that bothvaggregative and configurative SRs will be used in different ways, including forinstrumental, conceptual and legitimising purposes. A more positive view oflegitimising use than what is often considered was revealed by the interviews,suggesting that legitimising use can also be helpful to inform policy. Fourthly, two additional benefits of SR methods can be added in relation to whatresearchers can derive from these. This includes more systematic identificationof knowledge gaps and showing where the evidence is weak or contradictory. Bysearching for all available research and then applying strict criteria for whichstudies to include, SR can provide a clearer picture of what research is availableand not. Furthermore, the very process of conducting an SR means that theresearcher who performs it must be confronted with a wider range of literatureand be forced to study the quality of the studies in a way that is rarely done. Thiscan provide insights into the consequences of different method choices as well asto literature beyond the researchers’ own disciplinary focus