High Performance Linpack Benchmark: A Fault Tolerant Implementation without Checkpointing

The probability that a failure will occur before the end of the computation increases as the number of processors used in a high performance computing application increases. For long running applications using a large number of processors, it is essential that fault tolerance be used to prevent a total loss of all finished computations after a failure. While check-pointing has been very useful to tolerate failures for a long time, it often introduces a considerable overhead especially when applications modify a large amount of memory be-tween checkpoints and the number of processors is large. In this paper, we propose an algorithm-based recovery scheme for the High Performance Linpack benchmark (which modi-fies a large amount of memory in each iteration) to tolerate fail-stop failures without checkpointing. It was proved by Huang and Abraham that a checksum added to a matrix will be maintained after the matrix is factored. We demon-strate that, for the right-looking LU factorization algorithm, the checksum is maintained at each step of the computa-tion. Based on this checksum relationship maintained at each step in the middle of the computation, we demonstrate that fail-stop process failures in High Performance Linpack can be tolerated without checkpointing. Because no peri-odical checkpoint is necessary during computation and no roll-back is necessary during recovery, the proposed recov-ery scheme is highly scalable and has a good potential to scale to extreme scale computing and beyond. Experimen-tal results on the supercomputer Jaguar demonstrate that the fault tolerance overhead introduced by the proposed re-covery scheme is negligible