ESiWACE Scheduler development and support activities (D3.10)

About this document Work package in charge: WP3 Usability Actual delivery date for this deliverable: M35 Dissemination level: PU (for public use) Lead authors: Met Office (Met O): Matt Shin, Mick Carter, Dave Matthews Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC): Sandro Fiore, Alessandro D’Anca, Maria Mirto, Cosimo Palazzo, Giovanni Aloisio Other contributing authors: Max-Planck-Institut für Meteorologie (MPI-M): Reinhard Budich Barcelona Supercomputing Center (BSC): Domingo Manubens, Kim Serradell Reviewer: German Climate Computing Center (DKRZ): Philipp Neumann This document describes the work done in the context of the ESiWACE project on the topic Earth System Modelling scheduling (Task3.3) whose ultimate goal is to build a supported, user-driven community around the Cylc meta-scheduler for complex climate and weather suites on HPC systems and support advanced analysis experiments in the areas of data analytics and multi-site, multi-model experiments. This deliverable provides an update on this topic with respect to the Deliverable D3.9 - ESiWACE Scheduler development and support activities. The main focus of this effort relates to three aspects: Cylc developments (Task 3.3.2), Cylc Support Services (Task 3.3.3), and Scheduling Systems Development (Task 3.3.4). In particular, (i) the Cylc developments have followed the priorities agreed at a workshop on workflows organised by the IS-ENES2 project (Lisbon, September 2016) and include optimisations which supports the aims of large ensembles on Exascale platforms as anticipated in the long term, arising from workflow management and production-ready versions of the ESiWACE demonstrators, (ii) the Cylc support has addressed needs from the community regarding Cylc deployment and configuration at major data centres in the community, and, finally, (iii) the scheduling systems development has addressed major points on this topic, like architectural aspects and integrated scenarios, about large-scale analytics workflows for multi-model/multi-site data analysis. Here there are opportunities to build links to ESDM middleware from work-package 4. Finally, from a networking perspective, the 3rd Workshop on Workflows (funded by ESiWACE) has been organised over the last months and will be held in Brussels in September with 18 speakers presenting talks on several workflow topics, more than 40 registrants, 3 roundtable discussions and 1 training session.ESiWACE has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 67519