145 TFlops Performance on 3990 GPUs of TSUBAME 2.0 Supercomputer for an Operational Weather Prediction

AbstractNumerical weather prediction is one of the major applications in high performance computing and demands fast and high-precision simulation over fine-grained grids. While utilizing hundreds of CPUs is certainly the most common way to get high performance for large scale simulations, we have another solution to use GPUs as massively parallel computing platform. In order to drastically shorten the runtime of a weather prediction code, we rewrite its huge entire code for GPU computing from scratch in CUDA. The code ASUCA is a high resolution meso-scale atmosphere model that is being developed by the Japan Meteorological Agency for the purpose of the next-generation weather forecasting service. The TSUBAME 2.0 supercomputer, which is equipped with 4224 NVIDIA Tesla M2050 GPUs, has started operating in November 2010 at the Tokyo Institute of Technology. A benchmark on the 3990 GPUs on TSUBAME 2.0 achieves extremely high performance of 145 TFlops in single precision for 14368×14284×48 mesh. This paper also describes the multi-GPU optimizations introduced into the ASUCA porting on TSUBAME 2.0