Supplemental Material: MPI Derived Datatypes Processing on Noncontiguous GPU-resident Data

A number of efforts have been undertaken to integrate GPU functionality into an HPC environment, with modifications at the application, programming model, and library levels to account for a discrete GPU main memory space. Work related to MVAPICH [1], [2] is discussed in Section 2.3 of the Main Material. At the application level, algorithms that use both MPI and GPUs, such as Jacobsen et al.’s flow computation algo-rithm [3], are modified to allow efficient GPU computation, for example, changing the problem space partitioning to benefit GPU access patterns. MPI datatypes differ from these specialized data structures in that the datatypes ef-ficiently encode a subset of the data structures used, for use in communication and I/O routines. At the programming model level, the asymmetric dis-tributed shared memory model provides a single GPU address space across a cluster, while leaving GPUs aware of only their local memory space [4]. The consistency model is designed for and allows operating and processing on the shared address space in contiguous chunks with memory coherence; it would have to become more complex in order to enable the transfer and consistency of noncontiguous data or partial data within a contiguous buffer. Zippy [5] combines the message-passing and shared-memory models (based on Global Arrays) and provides a single address space for all GPUs in the cluster, using MPI as its backend. Zippy works specifically on multidi-mensional array-based data, so our work is applicable both to representing an area that needs to be transferred (such as noncontiguous array boundaries) and to subsequently packaging that data efficiently. At the library level, Distributed Computing for GPU Networks (DCGN) [6] extends MPI and utilizes signal