Data Handling Software Documentation and Roadmap - Deliverable D4.3

Managing data from modern high resolution weather and climate simulations is becoming problematic. Here we present a summary of some of the middleware software developed within ESiWACE2 to help manage the deluge of data during and after simulations. Middleware is software that lies between the computer, network, storage fabric and the scientific code that exploits that fabric. The objective of the middleware developed here is to exploit some of the scientific knowledge we have about our data to more efficiently use the underlying fabric. To that end, we present a description of two classes of software which we have developed and/or improved during ESiWACE2: (1) Tools for in-flight analysis of ensembles of simulation which ideally can support cross-ensemble calculations, avoiding the need for writing all variables all the time or at analysis frequency for all ensemble members to disk; and (2) Middleware for managing writing data to disk and managing the migration of data through storage tiers. The particular products we discuss are XIOS and it’s ensemble analysis tools, the Earth System Data Middleware (ESDM), the Near Line Data System (NLDS) and tools which exploit the Climate-Forecast Aggregation conventions (CFA-C, CFA-Python, CFA-CPP and CFStore). Not all of these pieces of software are feature complete, but for each we discuss their current status, where to find code and documentation, and their roadmap to the future.ESiWACE2 stands for Centre of Excellence in Simulation of Weather and Climate in Europe Phase 2. ESiWACE2 is funded by the European Union's Horizon 2020 research and innovation programme (H2020-INFRAEDI-2018-1 call) under grant agreement 823988