Increasing parallelism in climate models via additional component concurrency

Weather and climate models are severely limited by the strong scaling ability with respect to achievable throughput rates. For example, simulations of the German climate modelling initiative PalMod aim at covering multi-millennial time scales and therefore use very moderate spatial resolutions which allow for large time steps. At the other end global very high-resolution (i.e. kilometre-scale) simulations, such as performed within the ESiWACE Centre of excellence, demand for exascale computing due to the need for massive compute power and data capabilities. However, in this case simulations are limited to time scales of a few weeks. This is due to the small time steps that need to be used for physical and numerical stability reasons. In both the PalMod and the ESiWACE case the model throughput rate is limited by the scalability of the domain decomposition-based model parallelism. Radical approaches to improve scalability and parallel performance of Earth system models are therefore required. These approaches must be able to exploit the performance potential of available HPC systems. One of them is to introduce additional component-based concurrency similar to Balaji et al [1] and Mozdzynski and Morcrette [2]. For coupled Earth System models it is common to run different physical domains like atmosphere, ocean and ice independently and asynchronously between consecutive coupling time steps in separate groups of MPI-tasks. In this talk, we present an extension to this level of separation by introducing additional task parallelism within the same physical domain based on the approach of Mozdzynski and Morcrette. For this, we implemented a remote executing radiative transfer scheme for the ECHAM6 model. Here, we focussed on efficiency and flexibility of the communication between the radiation component and the dynamical core using the MPI-based YAXT communication library [3]. Preliminary results suggest that the additional communication overhead is negligible compared to the potential runtime improvement. The YAXT library is specially suited for bulk communication arising from single-domain task parallelism. This is due to the automatic generation of MPI datatypes that enable direct access to model data without requiring additional data copies or pack/unpack overhead to create messages. Besides these performance advantages YAXT also offers a simple way to generate diverse communication patterns. This allows exploring varying resource distributions and decompositions. It is furthermore planned to extend the implementation to other physical processes such as dust transport in ECHAM6-HAM to increase parallelism even further. [1] Balaji, V., Benson, R., Wyman, B., and Held, I.: Coarse-grained component concurrency in Earth system modeling: parallelizing atmospheric radiative transfer in the GFDL AM3 model using the Flexible Modeling System coupling framework, Geosci. Model Dev., 9, 3605-3616, doi:10.5194/gmd-9-3605-2016, 2016 [2] Mozdzynski and Morcrette, Reorganization of the Radiation Transfer Calculations in the ECMWF IFS, Technical Memorandum, ECMWF 2014. [3] https://doc.redmine.dkrz.de/yaxt/html/