Improving Multi-GPU Strong Scaling Through Optimization of Fine-Grained Transfers

Despite dramatic improvements in GPU and interconnect architectures, inter-GPU communication remains the most significant architectural bottleneck in multi-GPU systems. With hundreds of thousands of independent concurrently executing threads, maximizing interconnect utilization without degrading computational efficiency when strong-scaling HPC workloads is an open problem. In this dissertation, I will explore fine-grained peer-to-peer stores as the communication paradigm for improved multi-GPU strong scaling and propose three solutions to overcome the limitations of existing GPU and interconnect architectures to benefit from such transfers. First, I will detail PROACT, a joint compile and runtime system that transparently fine-tunes inter-GPU data movement for each application's needs, thus achieving the interconnect efficiency of bulk transfers at the programming simplicity of peer-to-peer stores. Next, I will demonstrate how GPS, a HW/SW memory management technique, employs selective page replication and proactive remote stores to improve read locality while conserving the interconnect bandwidth. Finally, I will discuss FinePack, a set of architectural enhancements to overcome the limitations of existing multi-GPU interconnects to perform small transfers efficiently.PhDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/172511/1/harinim_1.pd