BLIS on A64FX - Optimization Steps and Results

One of the key areas for enabling users to efficiently use an HPC system is providing optimized BLAS routines, allowing efficient dense linear algebra computations. Improving BLAS performance can help in utilizing the available compute resources on the largest supercomputers, which is however not a trivial endeavor and requires a lot of hardware specific optimizations. The BLAS-like Library Instantiation Software (BLIS) enables acceleration of Level-2 (matrix-vector) and Level-3 (matrix-matrix) BLAS operations through the insertion of only a few optimized microkernels. In June 2020 the Supercomputer Fugaku of the RIKEN Center for Computational Science in Japan became the #1 in the TOP500. In the Nov. 2020 TOP500 update, the performance was measured with 442.010 PFLOPS Rmax and 537.212 PFLOPS Rpeak. At the heart of Fugaku is Fujitsu's 48-core Arm-based A64FX processor which supports ARM's Scalable Vector Extensions - a new vector-length-agnostic SIMD extension. With a 512-bit SVE vector size, 2 SVE engines per core, and a clock frequency of 1.8 - 2.2 Ghz depending on the SKU, the A64FX is capable of delivering an impressive 2764.8 - 3379.2 GFLOPS of double-precision performance per node. Following up on previous efforts in porting BLIS to ARM SVE, this talk presents current numbers obtained on Isambard's A64FX nodes, where up to 92% of peak (2541 GFLOPS) DGEMM was reached with our BLIS version. The talk also gives an overview of the SVE microkernel implementation as well as which optimizations were necessary to reach those numbers - including leveraging some of the unique features of the A64FX processor