Using Structured Expert Judgment to predict a university ranking in the Humanities

This report is an assignment for the MOOC Decision Making Under Uncertainty: Applying Structured Expert Judgment. The goal of the exercise it to familiarize with the application of Structured Expert Judgment, the design and implementation of an elicitation and the reporting of findings. Structured expert judgment uncertainty quantification is a methodology developed by R. Cooke [3] at TU Delft and used in situations where decision making takes place despite a lack of information. This course is part of the Training Plan of the MSCA project Unveiling the Ecosystem of Science: A Contextual Perspective on the Many Roles of Scientists funded by the LEaDing Fellows Programme under the Marie Sklodowska-Curie grant agreement No 707404. The exercise aims at reporting a hypothetical ranking of Spanish universities in the field of Humanities. While Structured Expert Judgment is usually applied in fields in which there is a lack of data, in the particular example displayed here the starting point is different. We are analyzing a field in which there is a lack of consensus towards which data and how such data should be used to assess performance in these fields. The experts were directed to provide answers on the estimated rank a university should receive based on their knowledge and opinion. No indication as to which indicator should they used was given, hence the ranking should be considered as an overall ranking, comprising not only research performance, but also teaching and knowledge transfer. Having said this, as an exploratory exercise, the results should not be in any case considered as definite