#3

While High Performance Computing (HPC) systems continue to scale in volume of computing elements and overall computing powers, the performance/cost benefit of these systems is subject to their abilities to provide high reliability, availability, and transparency in utilizing the underlying computing resources. This is evidenced by a recent announcement [1] from Oak Ridge National Laboratory that their forthcoming machine, soon to be the world's fastest computer, will be a GPU cluster deployed across millions of cores. As such, fault tolerance has become a major concern in HPC, including GPGPU. In this paper, we propose a novel fault tolerance mechanism on GPUs and study the benefits of implementing such a mechanism in an HPC environment. This mechanism provides non-intrusive GPU check-pointing over an existing FT mechanism in an HPC system. It combines the checkpoint/restart mechanism, virtualization, and CUDA streams in order to achieve a low overhead solution. We provide a cost benefit model of our solution, analyze these results, and compare them against existing techniques. The mathematical model is presented and analyzed. The statistical analysis result shows that our mechanism increases the system resilience with a very reasonable cost. However, it slightly reduces system performance due to minimal additional overhead requirements. 1