Techniques for Modeling Large-Scale HPC I/O Workloads

Accurate analysis of HPC storage system designs is contin-gent on the use of I/O workloads that are truly represen-tative of expected use. However, I/O analyses are gener-ally bound to specific workload modeling techniques such as synthetic benchmarks or trace replay mechanisms, despite the fact that no single workload modeling technique is ap-propriate for all use cases. In this work, we present the design of IOWA, a novel I/O workload abstraction that al-lows arbitrary workload consumer components to obtain I/O workloads from a range of diverse input sources. Thus, re-searchers can choose specific I/O workload generators based on the resources they have available and the type of eval-uation they wish to perform. As part of this research, we also outline the design of three distinct workload genera-tion methods, based on I/O traces, synthetic I/O kernels, and I/O characterizations. We analyze and contrast each of these workload generation techniques in the context of stor-age system simulation models as well as production stor-age system measurements. We found that each generator mechanism offers varying levels of accuracy, flexibility, and breadth of use that should be considered before performing I/O analyses. We also recommend a set of best practices for HPC I/O workload modeling based on challenges that we encountered while performing our evaluation. 1