CSSI Element: Empowering Data-driven Discovery with a Provenance Collection, Management, and Analysis Software Infrastructure

Scientific breakthroughs are increasingly powered by advanced computing and data analysis capabilities delivered by high performance computing (HPC) systems. In the meantime, many scientific problems have moved to a level of complexity that the ability of understanding the results, auditing how a result is generated, and reproducing the important experiments or simulation results, is critical to scientists. Enabling such a capability in HPC systems requires an infrastructure for provenance data. In this project, we focus on provenance of jobs, users, nodes/hosts, files and the relationships among them. We developed to collect jobs from schedulers including Slurm and UGE (a component has been integrated into Dell HPC Omnia repository). We leverage jobstats and changelogs to gather file system provenance. We collected data from a 467-node cluster and conducted visual analytics. This project expects to improve the productivity of science in complex HPC simulation and analysis cycles