HemeLB on SuperMUC-NG performance assessment (POP2_AR_041)

This audit examined strong scaling of HemeLB on the SuperMUC-NG Lenovo ThinkSystem with up to 309,696 MPI processes (on 6,452 compute nodes) using a 21.15 GiB "circle of Willis" testcase. Although the code always ran correctly when sufficient memory was available, several compute nodes with notably inferior memory access performance were identified and needed to be explicitly avoided. Compared to the smallest configuration that could be run using 864 processes (on 18 `fat' compute nodes each with 768 GiB), 190x speed-up was delivered with 80% scaling efficiency maintained to over 100,000 processes for the simulation phase. Non-blocking MPI point-to-point message-passing ensures excellent communication efficiency, whereas load balance is somewhat variable but remains good at all scales. Computation scaling is excellent to over 50,000 processes before progressively deteriorating due to extra instruction overhead from local grid block surface to volume issues which are only partially compensated by improving IPC. HemeLB is an open-source lattice-Boltzmann code for simulation of large-scale fluid flow in complex sparse geometries such as those found in vascular networks. It is written in C++ using MPI by UCL and developed within the EU H2020 HPC Centre of Excellence for Computational Biomedicine (CompBioMed). The performance assessment report slides are complemented by an executive summary and a technical appendix