Direct numerical simulation of fluid turbulence at extreme scale with psOpen

The hybrid OpenMP/MPI code psOpen has been developed at the Institute for Combustion Technology, RWTH Aachen University, to study incompressible fluid turbulence by means of direct numerical simulations (DNS). For efficiency and accuracy psOpen employs a pseudo-spectral method, where the governing equations are solved partly in Fourier space. Hence, a pseudo-spectral method requires frequent transformations between real and Fourier space which is particularly challenging for massively-parallel setups. The frequently invoked Fourier transformation is a non-local operation and requires access to all data along a global grid line for each computational process. psOpen was improved by a new inhousedeveloped 3d-FFT library optimised to the special needs of pseudo-spectral DNS. For an optimal use of the hardware resources two techniques have been combined to reduce the communication time significantly. Firstly, the number of operations and the size of data to be transposed while computing a 3d-FFT has been reduced by integrating the dealiasing cut-off filter into the 3d-FFT. Secondly, the new 3d-FFT library allows to overlap communication and computation of multiple FFTs at the same time. The scaling performance of psOpen with three different grid sizes, namely 40963, 61443 and 81923, has been studied on JUQUEEN up to the full machine. © 2016 The authors and IOS Press. All rights reserved