GPUPool: A Holistic Approach to Fine-Grained GPU Sharing in the Cloud

As GPUs evolved into popular computing platforms in the cloud, GPU virtualization has become a highly desirable feature to lower datacenter operating costs. In this work, we explore fine-grained GPU sharing, which has a larger potential in improving hardware utilization compared to coarse-grained sharing adopted in Nvidia’s virtualization solutions. However, fine-grained sharing causes performance unpredictability due to workload-dependent interference, which makes it extremely challenging to provide quality-of-service (QoS) guarantees in the cloud. We present GPUPool, a GPU sharing system that unlocks the performance potential of fine-grained sharing at both systems and hardware levels. GPUPool predicts co-execution performance of arbitrary job combinations, and assigns compatible GPU jobs onto the same hardware to maximize hardware sharing. Our experimental results show GPUPool reduces the number of GPUs required to meet job- specific QoS requirements by 32% and 21.3%, compared to a system with no sharing and one with coarse-grained sharing respectively.M.A.S.2022-03-15 00:00:0