I/O Threads to Reduce Checkpoint Blocking for an Electromagnetics Solver on Blue Gene/P and Cray XK6

Application-level checkpointing has been one of the most popular techniques to proactively deal with unexpected fail-ures in supercomputers with hundreds of thousands of cores. Unfortunately, this approach results in heavy I/O load and often causes I/O bottlenecks in production runs. In this pa-per, we examine a new thread-based application-level check-pointing for a massively parallel electromagnetic solver sys-tem on the IBM Blue Gene/P at Argonne National Lab-oratory and the Cray XK6 at Oak Ridge National Labo-ratory. We discuss an I/O-thread based, application-level, two-phase I/O approach, called “threaded reduced-blocking I/O ” (threaded rbIO), and compare it with a regular version of“reduced-blocking I/O”(rbIO) and a tunedMPI-IO collec-tive approach (coIO). Our study shows that threaded rbIO can overlap the I/O latency with computation and achieve near-asynchronous checkpoint with an application-perceived I/O performance of over 70 GB/s (raw of 15 GB/s) and 50 GB/s (raw I/O bandwidth of 17 GB/s) on up to 32K pro-cessors of Intrepid and Jaguar, respectively. Compared with rbIO and coIO, the threading approach greatly improves the production performance of NekCEM on Blue Gene/P and Cray XK6 machines by significantly reducing the total sim-ulation time from checkpoint blocking reduction. We also discuss the potential strength of this approach with the Scal-able Checkpoint Restart library and on other full-featured operating systems such as the upcoming Blue Gene/Q