Message Passing for GPGPU Clusters: CudaMPI

Abstract—We present and analyze two new communication libraries, cudaMPI and glMPI, that provide an MPI-like message passing interface to communicate data stored on the graphics cards of a distributed-memory parallel computer. These libraries can help applications that perform general purpose computations on these networked GPU clusters. We explore how to efficiently support both point-to-point and collective communication for either contiguous or noncontiguous data on modern graphics cards. Our software design is informed by a detailed analysis of the actual performance of modern graphics hardware, for which we develop and test a simple but useful performance model. I. INTRODUCTION AND PRIOR WORK In 1968 Myer and Sutherland [1] asked a surprisingly prescient question: “How much computing power should be included in the display processor?”. Today, just as the decades