A qualitative evaluation approach for energy system modelling frameworks

Abstract Background The research field of energy system analysis is faced with the challenge of increasingly complex systems and their sustainable transition. The challenges are not only on a technical level but also connected to societal aspects. Energy system modelling plays a decisive role in this field, and model properties define how useful it is in regard to the existing challenges. For energy system models, evaluation methods exist, but we argue that many decisions upon properties are rather made on the model generator or framework level. Thus, this paper presents a qualitative approach to evaluate frameworks in a transparent and structured way regarding their suitability to tackle energy system modelling challenges. Methods Current main challenges and framework properties that potentially contribute to tackle these challenges are derived from a literature review. The resulting contribution matrix and the described application procedure is then applied exemplarily in a case study in which the properties of the Open Energy Modelling Framework are checked for suitability to each challenge. Results We identified complexity (1), scientific standards (2), utilisation (3), interdisciplinary modelling (4), and uncertainty (5) as the main challenges. We suggest three major property categories of frameworks with regard to their capability to tackle the challenges: open-source philosophy (1), collaborative modelling (2), and structural properties (3). General findings of the detailed mapping of challenges and properties are that an open-source approach is a pre-condition for complying with scientific standards and that approaches to tackle the challenges complexity and uncertainty counteract each other. More research in the field of complexity reduction within energy system models is needed. Furthermore, while framework properties can support to address problems of result communication and interdisciplinary modelling, an important part can only be addressed by communication and organisational structures, thus, on a behavioural and social level. Conclusions We conclude that the relevance of energy system analysis tools needs to be reviewed critically. Their suitability for tackling the identified challenges deserves to be emphasised. The approach presented here is one contribution to improve current evaluation methods by adding this aspect