Improving GPGPU concurrency with elastic kernels

Each new generation of GPUs vastly increases the resources avail-able to GPGPU programs. GPU programming models (like CUDA) were designed to scale to use these resources. However, we find that CUDA programs actually do not scale to utilize all available resources, with over 30 % of resources going unused on average for programs of the Parboil2 suite that we used in our work. Current GPUs therefore allow concurrent execution of kernels to improve utilization. In this work, we study concurrent execution of GPU kernels using multiprogram workloads on current NVIDIA Fermi GPUs. On two-program workloads from the Parboil2 benchmark suite we find concurrent execution is often no better than serial-ized execution. We identify that the lack of control over resource allocation to kernels is a major serialization bottleneck. We pro-pose transformations that convert CUDA kernels into elastic ker-nels which permit fine-grained control over their resource usage. We then propose several elastic-kernel aware concurrency policies that offer significantly better performance and concurrency com-pared to the current CUDA policy. We evaluate our proposals on real hardware using multiprogrammed workloads constructed from benchmarks in the Parboil 2 suite. On average, our proposals in-crease system throughput (STP) by 1.21x and improve the aver-age normalized turnaround time (ANTT) by 3.73x for two-program workloads when compared to the current CUDA concurrency im-plementation