Maintaining Communication at Scale with OpenSHMEM

As the dawn of the exascale era arrives, high-performance computing (HPC) researchers continue to seek parallel-communication models that perform well on increasingly large distributed systems. The SHMEM (Shared Hierarchical Memory) family of parallel programming libraries has been under development over the last three decades by a community of researchers, government organizations, and corporations. SHMEM has a variety of implementations that have recently been expanded to distributed-memory parallel-computing clusters. The OpenSHMEM project is one of these efforts and has emerged as a standardized application-programming interface that is designed for portability and support of the partitioned global address space (PGAS) model. To investigate the performance characteristics of SHMEM, this research focuses on developing, deploying, and collecting metrics of two variants of the 2D fast Fourier transform algorithm, as well a modified version of the Horovod framework for distributed machine learning. A comparison to OpenMPI's message-passing interface (MPI) methods will be conducted as a point of reference. We show that in a 2D FFT application that is communication-bound by a transpose stage, OpenSHMEM's collective operations outperform that of MPI RMA. On this 2D FFT application, we demonstrate efficiencies of 0.81, 0.80, and 0.93 at largest node counts on PSC Regular Memory, PSC Extreme Memory, and NERSC Perlmutter, respectively