Optimizing the research and innovation policy mix: The practice and challenges of impact assessment in Europe

Executive summary BACKGROUND European challenges and the importance of impact assessment As a follow-up to the Lisbon strategy, the European Commission (EC) presented the ‘Europe 2020’ stratey. The main vision underlying this strategy is to turn Europe’s socio-economic development and progress into a smart, sustainable and inclusive economy delivering high levels of employment, productivity and social cohesion. Public European and, even on a larger scale, national (and regional) RTDI funding plays an important role in making ‘Europe 2020’ ambitions a reality. This role relates to the reality of market failures, which prevent the private sector from investing in research at a socially-optimum level. The Europe 2020 Flagship Initiative Innovation Union states that ‘progress’ should be measured at the European Council level for monitoring and intervention purposes, and that close monitoring of the impacts of the various policy measures is required. The realisation of the Europe 2020 objectives is hence the shared responsibility of the European Institutions and national and regional authorities of the Member States. Impact assessment and RTDI policy design in Europe Evaluation activities of many Member States (and of the Community) have reached a certain level of maturity. The need to strengthen the impact of evaluations on policy and policy design remains a major concern, and is something which requires closer interaction between policy decision makers, those entrusted with policy implementation, and those undertaking evaluation activities. This proves to be a daunting task, since ‘the policy cycle is not closed’. One of the most important factors which hinders the utilization of IA in policy optimization is the lack of (early) involvement of government administrations/agencies, politicians and the programme management agencies/organisations in IA studies. Methodologically, there are also some serious challenges, concerning the take-up of IA findings and results into ‘ex-ante’ policy formulation. This leads to a ‘missing link’ between the impact assessment and the take-up of its results by policy makers. There are often no feedback loops between policy formulation, implementation, and optimisation. Hence, there is a need for a Europe-wide exchange of IA practices and experiences. The OMC-net project The OMC (Open Method of Coordination) generally rests on soft law mechanisms such as guidelines and indicators, benchmarking and sharing of best practice. Taking into account initial expertise, experience and expectations, and the main strengths of public administrations and agencies in terms of policy formulation and support, the primary goal of underlying OMC-net project has been: (i) to increase the usefulness of IA in the context of RTDI policy formulation, or in other words, making sense of IA for policy making, and (ii) to provide a useful framework facilitating communication between policy makers and different stakeholders in the policy making process. The focus of this work is on ex-post IA and its role in RTDI policy making, particularly in the areas of ‘public funding of private RTDI’, ‘industry-science linkages’, and the ‘performance-based funding of universities’. In order to improve feedback from ex-post IA exercises to policy making, more attention needs to be paid to: (i) accommodating the shift in emphasis from outputs to results and impacts; (ii) accommodating the shift from single measure ex-post IA to policy mix ex-post IA; (iii) accommodating the dual role of ex-post IA – learning and accountability; and (iv) embedding lessons learning. These elements can be crucial for closing the policy cycle and the take-up of the results of IA exercises in policy development. The OMC-net project reflects on how to appreciate whether the policy measures are heading the right way, before, during and after their implementation, and the role of IA in this context. It pays particular attention to mutual learning, harmonisation, and usefulness for policy design. Four horizontal topics with respect to IA are targeted:  The practice of IA in the policy cycle;  The governance of the IA process to increase the use of IA as a tool for policy making;  The barriers and critical success factors in the transfer of best IA practices;  Methods and indicators for performance-based funding of research. KEY FINDINGS The main findings presented below are derived from this final report of the OMC-net project. They have been thoroughly discussed and debated at the Belgian EU Presidency Conference ‘Common Impact Assessment for Optimising the Policy Mix’ held on December 8th 2010 in Brussels. The ‘practice’ of Impact Assessment in the policy cycle During the course of the project implementation, various literature sources on IA and policy design and evaluation have been reviewed (European Commission, OECD, etc.). When combined with the analysis of a large number of case studies by the OMC-net partners, a good insight into the use and practices of IA in Europe today was obtained:  There is an increased interest and use of IA in policy making towards RTDI in Europe, albeit often on an ad-hoc and exceptional basis;  Heterogeneous visions and terminology for IA exist; there is thus a ‘space’ for further conceptual and practical harmonization in Europe while respecting the ‘no one size fits all’ principle;  Minimal attention is paid to the broader societal impact of public RTDI funding, although policies increasingly target wider societal ‘grand’ challenges. This is largely due to a lack of suitable indicators and data. But this may not be used as an ‘excuse’ for engaging in impact assessment;  Minimal attention is given to additionality and the interaction between different forms of additionality;  The policy making cycle is not ‘closed’: policy uptake of outcomes of IA exercises is suboptimal. Early involvement of policy makers is essential. In IA of RTDI policy measures the focus remains on the ‘mere’ measurement of outputs, outcomes, impacts, goal attainment, and in few cases, on additionality. Too little attention is paid to policy/strategy development and project implementation efficiency and how the IA can contribute hereto. The policy cycle remains ‘unclosed’. Less than 50% of the IA studies requested by government administrations/agencies really involve policy makers and/or programme management in an early stage. Similar concerns apply to the involvement of the ‘beneficiaries’ of specific measures. The broader policy design is also not an objective in more than 50% of the IA exercises, and its influence on and relationship with other policy measures (e.g. in terms of re-design or merger) are not considered. This reveals the minimal attention paid to the broader policy mix, and thus the lack of a systemic approach and perspective in the IA exercises. There are serious methodological and technical challenges. It is difficult to approach the complexity of the issue and to find appropriate indicators to measure the different impacts (scientific, technologic, economic and societal). It is also challenging to address the causality between public funding of RTDI policy measures and their impacts. The appropriability of indicators is case dependent. Societal impacts appear especially difficult to measure. Progressing in the development and usage of ‘appropriate’ indicators for measuring the socio-economic impact can help the formulation of the ‘logic intervention model’. Moreover, this will enhance the usefulness of IA as a tool for policy makers since policy making still largely becomes evidence-based (e.g. the ‘Europe 2020’ strategy). Governance of IA projects Governance of IA relates largely to the way IA-exercises are designed and managed throughout their implementation:  The design and governance of IA play a critical role in the usefulness of IA for policies improvement;  Methodological and governance choices are clearly driven by the objectives of the IA, and the specificity of the policy area and the RTDI measures - the setup is ‘situational’;  Sufficient attention should be paid to the ‘design of IA processes’ and to the role of the programme manager (agency or administration);  Impact assessments should be an ongoing ‘learning process’ which requires a smart broad involvement of different stakeholders;  Sufficient attention should be paid to the ‘follow-up of the IA results’, to timing and to how results are communicated to policy makers. Impact assessments, when carried out properly, can be a source of legitimacy for policy makers and can be useful as a policy ‘learning’ tool. However, to be effective, careful attention needs to be paid to ‘product factors’ related to the results of findings of the IA exercise (operational learning, policy feedback, system impact, presentation of information, and scope) and to ‘process factors’ related to the usefulness of IA created during the process of its conduct (timing; programme objectives; policy-makers’ expectations; policy-makers’ engagement; trust and compliance of all stakeholders and dissemination). It is essential that IA is considered early on in the programme design phase (ex-ante) as a process and not only as an event in its life cycle. Impact assessment should be an integral part of a learning process in parallel to the other activities of the RTDI programme. Impact assessment should also be ‘broad’ enough, i.e. focused not only on the outcome of the programme, but also included in the operations of the agencies/programme management. With a clear description of the context, objectives, needs and issues related to the study, a high quality ‘terms of references’ (ToR) is necessary for a high quality IA. Timing is a central planning and design factor - from the time of the study and its different stages to the point where its impacts begin to materialize. Innovation impacts typically take some time to become observable (in terms of years). Impact assessment should be conceived as an open and ongoing process, based on the accumulated learning effects achieved through sequencing various studies over time. The learning process regards evaluation can be strengthened through international benchmarking and review of IA studies, accounting for those differences between national contexts which must be taken into consideration. Consultation processes or discussion panels (external peer reviews) may be very useful. Trust and compliance are other useful factors of usefulness. Involving stakeholders from the outset by creating transparent objectives, scope and methodologies is also essential. Evaluation is as much a social process as in any other business operation. Formulation of the policy recommendations is an essential part of the IA exercise which determines the quality and relevance of the information regarding policy feedback. The background and expertise of the evaluator should be part of the selection criteria for evaluators. Language and format policy recommendations must be accessible for the dissemination process. In order to establish an effective platform for utilization or usefulness, evaluation and IA should be linked to decision points at policy, agency or programme level. An agency should involve itself in turning evaluation results into action. These actions should be carefully planned in order to support stakeholders (including beneficiaries) to exploit the results of evaluations and IA. Barriers and critical success factors for transferability of impact assessment practices  RTDI policy instruments are not always ‘open’ in terms of design, context, indicators, and methods, which hinder transferability and mutual learning.  The design, implementation and modalities of RTDI policies differ throughout countries and regions, which makes transferability of IA approaches quite challenging.  The transferability of IA exercises depends on the presence of the context specific critical success factors which explain the diversity of RTDI policy measures.  Due to the diversity of policy measures which are devised in specific contexts, it is necessary to design IA exercises in a variety of different ways. The conditions required in order to share future knowledge and experiences related to IA practices and results also have to be considered. In the context of this OMC-net study, it became clear that in order to share knowledge and experiences, IA practitioners and analysts have an enormous responsibility – equivalent to that of policy makers on various levels. Impact assessment exercises should be well documented, in such a way that the approach (and ideally the results) can be reproducible (methodological details are thus needed). Policy makers should have the responsibility for documenting and describing the particular policy measures in order to understand their role in the design of the IA process. Likewise, they should choose analysts/evaluators who are known for their transparent and well-documented IA exercises. To disseminate successfully IA practices in Europe (assuming that this is a legitimate objective), openness and transparency and ‘situational’ thinking should be required. An European IA protocol could be very useful in this respect. Methods and indicators for performance-based funding of research  Even though performance-based funding sounds a rather homogenous concept, it is applied differently. Therefore, both the specific context and culture need to be taken into account when assessing the impact of performance-based funding policy instruments.  To increase the acceptability of performance-based research funding schemes, the targeted actors need to be appropriately involved in the process of designing such schemes.  The impact of performance-based funding can only unfold after a longer time period. This and any unintended consequences have to be taken into account during IA planning.  The use of a single quantitative and qualitative assessment method has its flaws. It is therefore necessary to implement combinations of various quantitative and qualitative methods so as to provide a more adequate (holistic) overview of the impact of performance-based funding of research. But even when doing so, it has to be taken into account that many limitations still remain for the assessment of performance-based funding of research. Increasingly, performance-based funding is an important policy choice made in the context of discussions on competitive versus block funding models. Despite the similarities in the objectives and rationales of performance-based funding schemes, the design of the respective policy instruments vary from country to country, reflecting specific political and regional contexts. Because of the short duration of performance-based funding schemes, there is only limited evidence regarding the effects and impacts of such measures. With regard to recommendations relating to IA of performance-based funding, some pitfalls need to be avoided. They include the need to take into consideration national characteristics and the multi-tasked nature of higher education systems and/or the systems associated with performance-based funding. When designing and implementing impact assessment exercises, incentives regarding research efforts have to be balanced so as to prevent suppressing the intrinsic motivations of academics or streamlining their activities towards certain activities, irrespective of their particular talents. It should also be borne in mind that these schemes can introduce an abrupt and complete departure from the previous system unless ‘grind-in’ possibilities are introduced for the affected institutions and personnel in order to ensure the acceptance of the scheme by the academic community. Despite the merits of performance-based funding schemes, it is important to recognize that many limitations and problems still remain. Therefore, some main conclusions for the policy level is that they are aware of the diverse limitations of performance-based research funding schemes; to address them as far as possible and to recognize that further work (particularly on effects and impacts of the schemes implemented) has to be done in order to address effectively such limitations. LESSONS LEARNED AND THE WAY FORWARD Where do we go from here? When it comes to priority areas for further development with regard to the use of IA, three areas for prioritisation of future work have been explored. They include: further methodological development and learning, development of indicators for IA, and the take-up of usefulness of evaluation and IA outcomes in policy making. In that respect, the following recommendations have been identified:  A new conceptual framework and way of thinking on how to ‘close the policy cycle’ should be developed There is still a long way to go for ex-post IA in RTDI policy making, particularly concerning the setup and governance of IA exercises (process and product factors) to ensure smooth execution of IA exercises and maximum take up of results for policy improvement. The latter is what we earlier coined as ‘closing the policy cycle’. In this perspective a new conceptual framework and way of thinking on how to close the policy cycle is urgently needed.  There is a need to establish a permanent EU-wide IA community and to bring it closer together in order to share experiences This OMC-net project clearly illustrated the usefulness of exchanging practices and experiences by countries, funding agencies and IA practitioners. There is a strong plea to continue these kinds of harmonization efforts in future, and to take into account more carefully the starting points of the various actors and their level of experience with IA. An EU wide IA community could ensure systematic coordination and exchange of experiences (a pool of knowledge). Such an IA community could be linked to the existing European Evaluation Network which is currently moderated and facilitated by the European Commission. One advantage of a pool of IA practitioners is that national governments and agencies could involve members of this pool as ‘peers’ in setting up and implementing their own IA. Such a community, if clearly identifiable, could help the further introduction of IA particularly in those countries where there is less experience and clearly a greater need for impact assessments. It seems to be challenging to agencies when they need to select the right experts to support the design and implementation of an impact assessment. As an analogy to the concept of the ‘peer review’, one could think of creating regular peer reviews of impact assessments in different countries and regions. For example, ‘peers’ - who have sufficient demonstrational expertise and commitment - could be involved when setting up and/or carrying out an impact assessment exercise.  Paying attention to the ‘design’ and the ‘governance’ of IA is essential, where a systemic (‘holistic’) approach is advisable The OMC-net project has clearly illustrated numerous shortcomings in the design and governance of evaluation and IA studies in Europe. Practical ‘tips’ and ‘tricks’ will help funding agencies and other practitioners to improve the setup of IA exercises and also to increase the chances of better uptake of the findings into new policy development. However, the results also show that there is a need to better understand (both conceptually and methodologically) what it means that RTDI policy is increasingly aiming to tackle the ‘grand challenges’. Framing and measuring the societal impacts thus requires a new methodological and indicator framework, and a higher level of analysis than the individual policy measures allowed. This requires a more holistic programme perspective, and even an RTDI policy measures cluster perspective. Adequate metrics are also an important element herein. This also implies that multiple impacts - economic, scientific, technological and societal - have to be considered on all EU governance levels.  Ex-post IA starts ex-ante; mid-term IA should check, whether we are still ‘on course’ It is important to ex-ante identify the expected effects and impacts of a specific policy measure. It needs then to be considered ex-post whether the expected effects and impacts have been realized. One should not neglect the importance of mid-term assessments, which provide an opportunity to adjust the ongoing policy measures towards the desired goals, results and impacts. There is no doubt that mid-term assessments cannot fully appreciate the (potential) outcomes/impacts due to the time lag between policy intervention and resulting impacts. It can, however, provide early signals about the likelihood of achieving the desired impact.  Policy makers and administrators should be trained in IA The practice and experience gained under this OMC project clearly highlights the importance of the training of policy officers in funding agencies. The most important element for the closure of the policy cycle is the human factor. Programme managers, policy officers and policy makers need to understand how an IA project works, and how the recommendations provided can be turned into reality. Impact assessment practitioners and programme managers in funding agencies need to know how to create the necessary ‘buy in’ in order to mobilize the drivers for policy improvement at all echelons after assessment. Investment in training is essential. The perspective of the OMC-net participants Considering the lessons learned from this OMC-net project, it is important to take into account the initial context in terms of experiences, expertise and interests of the participants. The IA expertise of the project partners at the beginning of the project was moderate, their experience was limited and their intere