Open Science in research assessment. An overview of quantitative and qualitative approaches

Research assessment reform is an urgent priority of European research policy. While initially closely tied to the advancement of Open Science practices, this objective has grown into a larger movement that addresses many different shortcomings of dominant evaluation practices. European University Alliances are directly concerned by this policy objective and are seen as possible testbeds for reform. EUTOPIA has also committed to implement a framework policy for research assessment and Open Science through the EUTOPIA-TRAIN project. With this paper, we aim to present practical methods and indicators addressing Open Science that are being discussed and piloted at research performing organisations across Europe. We hope to stimulate the institutional transformation of our own alliance and otter universities through the provision of the practical ideas covered in this report. The report builds on an earlier background note of EUTOPIA in which we have explored the political momentum in the area of research assessment and its implications for European Universities. This report includes sections covering an overview of initiatives and declarations which seek to reform research assessment, including DORA, the Leiden Manifesto and others. Our summary focuses on the role of quantitative and qualitative assessment in these initiatives and declarations. quantitative indicators and metrics that could be used to measure Open Science activities and outputs. Our overview is based on existing source documents from various national and international frameworks. We also provide a summary of the commonalities of these frameworks- as well as challenges when implementing quantitative approaches. qualitative methods for assessment. Our overview is largely based on the emerging experience with narrative CVs, but other measures taken, for instance contextualising achievements and outputs, as well as contravening biases in qualitative assessment are addressed, too. the underlying infrastructure and (open) information systems for assessment that have been discussed and proposed elsewhere to provide an comprehensive, up-to-date, and accessible evidence base for assessments. We also summarise briefly the obstacles one encounters when assessing the FAIRness of research data