D4.1 Advanced software stack for Earth system data

An exascale simulation requires extreme-scale I/O. Existing workflows will not scale to this extent, particularly as an exascale simulation is likely to involve a smaller number of individuals carrying out large simulations and a larger number of individuals carrying out analysis of simulation products. Such analysis is bound to include data comparison from simulations carried out in multiple locations and stored on a range of media. ESiWACE addresses the entire workflow with software solutions addressing a range of targets, from simulation, through analysis, to information management across institutions. Most of the effort has been devoted to the simulation problem with work on both improving I/O from simulations (using the Earth System Data Middleware, ESDM) and on ensemble data analysis (using the X I/O server, XIOS). The ESDM provides relaxed I/O semantics and manages the heterogeneous storage available in a data centre by mapping datasets dynamically to (possibly multiple) storage locations, addressing both simulation and analysis use cases. This document provides a high-level descriptions of that software stack, including full documentation and information about how to enable ESDM support for existing NetCDF applications and utilise the ESDM API for software development. As part of the development that resulted in the ESDM architecture described and documented here, the ESDM code was hardened, and the I/O path has been investigated for many use cases and optimised. The performance measurements indicate that performance can be close to optimal for large-scale data access. Several additional features had been explored and prototyped that bear a high potential for large-scale performance improvements and co-design with vendors. These include new storage backends for S3, Motr, KOVE, PMEM-IO, and IME, which have been developed to bypass the POSIX API and accelerate performance. Ensemble data analysis can be carried out using existing tools (CYLC and XIOS), albeit with customisations that are model dependent. The necessary software can be built using NetCDF directly, or with the ESDM