Automated application-level checkpointing of MPI programs

Because of increasing hardware and software complexity, the running time of many computational science applica-tions is now more than the mean-time-to-failure of high-peformance computing platforms. Therefore, computational science applications need to tolerate hardware failures. In this paper, we focus on the stopping failure model in which a faulty process hangs and stops responding to the rest of the system. We argue that tolerating such faults is best done by an approach called application-level coordi-nated non-blocking checkpointing, and that existing fault-tolerance protocols in the literature are not suitable for im-plementing this approach. In this paper, we present a suitable protocol, and show how it can be used with a precompiler that instruments C/MPI programs to save application and MPI library state. An advantage of our approach is that it is independent of the MPI implementation. We present experimental results that argue that the overhead of using our system can be small.