Update of ESiWACE Application Software Framework prepared for the demonstrators (D3.2)

Abstract Software implementations of modern climate and numerical weather prediction models and their execution workflows require many pieces of software to be correctly installed and configured. Various mathematical, input/output, parallelisation and other software libraries are necessary to build and run executables of the model; plotting, statistics, file format conversion and other packages are required for post-processing; scripting language interpreters and filesystem utilities are essential for controlling different stages of the execution workflows and gluing them together. All these generate long lists of software dependencies that have to be resolved and accounted for during the deployment of the complete software stack required for numerical experimentation. This process is further complicated by the fact that the models are research tools and hence are under continuous development. It is often the case that introduction of changes in the model’s source code becomes an essential part of a research workflow. Thus, the software environment for the research workflow does not have to only fulfil the runtime requirements of the model execution workflow but also the requirements posed by software development needs. On top of that, given the resource requirements of some scientific projects, a single machine is not always enough to accommodate all required model runs. Thus, it becomes mandatory to make the software stack portable to whatever supercomputing environment becomes available for the numerical experiments. This work aims to systematise requirements of a typical climate modelling workflow to the software environment, to identify common problems on the way to meeting them, and attempts to remedy the situation by providing recommendations on what solutions can be applied to make the preparations of the software environment for Earth system modelling easier. There are no indications that the requirements to the software environments of the upcoming extreme scale HPC will decrease. On the contrary, the main challenges related to the increasing amount of input and output data of the models, workload balancing, and fault tolerance (see ESiWACE deliverable D3.8), will be exacerbated, requiring more and more sophisticated solutions, the implementation of which will inevitably lead to the further complications of the already difficult problem of the software dependencies resolution. At the same time, most experts assume that the basic concepts underlying the existing approaches to the software deployment and maintenance will be inherited in the next generation of HPC solutions. Thus, the investment into the development of the automatization tools for software installations promises long term benefits for the whole HPC community. About this document Work package in charge: WP3 Usability Actual delivery date for this deliverable: 5 March 2018 Dissemination level: PU (for public use) Lead author: Max Planck Institute for Meteorology (MPI‐M), Sergey Kosukhin, Reinhard Budich Other contributing partners: Barcelona Supercomputing Center (BSC), Kim Serradell, Oriol Mula-Valls The University of Reading (UREAD), Grenville Lister Rewier: German Climate Computing Center (DKRZ), Chiara BearzottiThis project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 67519