Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach

The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning