Portfolio Evaluation: FWF International Programmes

The present evaluation focuses on the international programme portfolio of the Austrian Science Fund (FWF). We assess the appropriateness of the portfolio as well as of its design and management. In addition, we trace evidence for the impact of the international programmes. The evaluation provides results that are relevant for FWF’s strategic discussions on future programme design. In order to address the evaluation questions as specified in the terms of reference, we designed a multi-method approach combining quantitative and qualitative elements: bibliometrics, altmetrics and various descriptive statistics, on the one hand; interviews, a survey and a focus group on the other. In order to assess the additionality and impact of international programme support, we used statistical matching techniques to construct a comparison group of FWF-supported stand-alone projects. FWF’s international programmes mobilise and, thus, support a segment of the Austria-based research community that is able to produce high-quality output with international colleagues. Principal investigators in international programme projects are well-established scientists, often in Professor positions and with unlimited contracts (young researchers use different programmes for international cooperation). Many of them have experience with other international cooperation support programmes and have had one or several long-term mobility experiences. However, the geographic patterns of cooperation in international programmes are different from PIs’ mobility experiences. We conclude that FWF’s international programmes contribute significantly to the internationalisation of science in Austria. Additionality to other national as well as European funding schemes is given. Particularly the thematically open funding of research projects is a unique aspect of FWF’s portfolio. International cooperation is also widely practiced in FWF-supported stand-alone projects. However, the geographies of cooperation are slightly different. Also, the intensity of the collaboration is higher in the international projects. Our bibliometric analyses show a consistently higher impact of publications in international programme projects compared to stand-alone projects. The sustainability of the cooperation triggered by international programmes can only be indirectly assessed by now (most international projects started in recent years). A majority of survey respondents indicates that they continued their cooperation beyond the I project. The analysis of publication histories suggests that I-projects either help to continue existing collaboration patterns or to induce new ones. Beneficiary satisfaction with programme management and related processes is high. The various instruments are in demand and, although success rates are somewhat low, they are still considered adequate by most. The beneficiaries are also satisfied with the international programme portfolio. 40% of our survey respondents consider their need for international cooperation support is met by current programmes, 49% see it partially met. Thematically open, bottom-up support to multilateral cooperation is demanded as an expansion of the portfolio. There is also a majority asking for support to networking. With the help of the evidence collected, we facilitated discussions on future scenarios for the international programmes. On this basis, we suggest a combination of: continuation of the portfolio incl. selected country strategies without fragmenting into too many programmes; FWF to continue to push for multilateral bottom-up funding e.g. in a European context; the exploration of an FWF-funded networking scheme. We also propose to consider the age structure of I project PIs and to continue monitoring aspects where data availability is currently still limited, like citation impact and sustainability. Here are the FWF's comments on the evaluation report: https://doi.org/10.5281/zenodo.119456