High-Throughput In-Situ Workflows for Molecular Dynamics Simulations

Next-generation computing systems are expected to have dramatically higher performance than current systems. However, I/O bandwidth and parallel file system capacity will not grow at the same rate. This I/O bottleneck constitutes a major challenge for the analysis of data-intensive high-performance computing (HPC) simulations. This work targets a critical application for future exascale systems: molecular dynamics (MD) simulations studying the classical time evolution of a molecular system at atomic resolution. We enable the development of new comprehensive workflows for MD simulations in which HPC meets data analytics empowered by machine learning (ML). In situ processing, combined with in-memory computing, has emerged as a solution to overcome I/O bottlenecks in HPC systems. How to do this efficiently from the infrastructure and human resources perspectives is still a challenging open problem. We address this challenge by designing and developing A4MD: a workflow engine that enables better scientific knowledge discovery in the domain of MD. A4MD features plug-and-play workflow composition of reusable building blocks; transparent in situ annotation capability for user-defined methods; simulator-agnostic runtime trajectory analysis; support for stopping, starting and restarting simulations; ensemble workflow patterns optimized for HPC environments; and an overall design suitable for ML analysis. A4MD empowers scientists to integrate simulation and advanced analytics into complex workflows for automatic runtime detection of changes in structural and temporal molecular properties. This knowledge of molecular structures' transformations at runtime can be used to steer simulations to more promising areas of the simulation space, identify the data that should be written to congested parallel file systems, and index generated data for retrieval and post-simulation analysis