D2MA: Accelerating Coarse-Grained Data Transfer for GPUs

To achieve high performance on many-core architectures like GPUs, it is crucial to efficiently utilize the available mem-ory bandwidth. Currently, it is common to use fast, on-chip scratchpad memories, like the shared memory available on GPUs ’ shader cores, to buffer data for computation. This buffering, however, has some sources of inefficiency that hin-der it from most efficiently utilizing the available memory resources. These issues stem from shader resources being used for repeated, regular address calculations, a need to shuffle data multiple times between a physically unified on-chip memory, and forcing all threads to synchronize to en-sure RAW consistency based on the speed of the slowest threads. To address these inefficiencies, we propose Data-Parallel DMA, or D2MA. D2MA is a reimagination of tradi-tional DMA that addresses the challenges of extending DMA to thousands of concurrently executing threads. D2MA de-couples address generation from the shader’s computational resources, provides a more direct and efficient path for data in global memory to travel into the shared memory, and introduces a novel dynamic synchronization scheme that is transparent to the programmer. These advancements allow D2MA to achieve speedups as high as 2.29x, and reduces the average time to buffer data by 81 % on average