Report of Appliances Available for Testing - Deliverable D4.2

This deliverable reports on the two storage technologies from two vendors, DDN and Seagate, which were designed to work with the Earth System Data Middleware (ESDM) - details of which were reported in Deliverable D4.1 “Advanced software stack for Earth system data”[ESiWACE2 D4.1]. The expectation was that various storage appliances would have support for ESDM and could be deployed with ESDM at the weather and climate sites as ESDM would become available. Seagate demonstrated the CORTX Motr (formerly Mero) software installation being capable of working with commodity hardware, with the ability to be installed at the German Climate Computing Center (DKRZ) and other centres in the future as per their requirements. This is a change in strategy from Seagate for CORTX Motr. At the beginning of the project, the plan was to have an appliance based on Mero – as Mero was closed source software. However, during the time frame of ESiWACE2, Mero was open sourced as CORTX Motr – making it feasible to be deployed very widely on commodity hardware. Showcasing CORTX Motr’s capability to work on commodity hardware working with ESDM, we believe is a powerful message for the community rather than providing just a “closed” appliance. This deliverable hence takes that approach. DDN demonstrated the relevance and the efficiency of the Earth Systems Data Middleware (ESDM) developed during the ESiWACE project on state-of-the-art industrial storage technologies. DDN implemented a sophisticated integration of ESDM with the flash native, burst buffer solution product of DDN’s portfolio, Infinite Memory Engine (IME). As part of the integration ESDM now supports the IME native interface that bypasses the FUSE, allowing to obtain the maximum performance of IME. The code of the integration is available and accessible to everyone through GitHub. Using IME, DDN was able to produce performance results significantly beyond the state of the art in term of hardware / software cooperation. Furthermore, ESDM provides performance in the range of the low-level API while still offering a higher level of semantic to the end-user. The results were obtained within one of DDN’s labs using 5 IME appliances and 10 clients. With an aggregated peak I/O bandwidth of 100 GB/sec, this setting is representative of the core of the HPC / technical computing market. In addition to the lab set-up and the integration effort, DDN has led more theoretical and research-oriented work. In respect to the Active Storage task, DDN provided a complete design architecture and implemented a proof of concept (PoC). The prototype is expected to be ready within the Q1 of 2023. The PoC code of the active storage is available on GitHub