Scalable HPC Workflow Infrastructure for Steering Scientific Instruments and Streaming Applications

Linking scientific instruments to exascale machines and analyzing the large volumes of data produced by the instruments requires workflow infrastructure that scales along many dimensions. In this white paper we generalize this problem to include control systems, analysis of data produced by supercomputers, computational steering and data assimilation. The requirements of distributed computing problems which couple HPC and streaming data, are distinct from those familiar from large­scale parallel simulations, grid computing, data repositories and orchestration, which have generated sophisticated software platforms. Our analyses points to new research directions for a scalable infrastructure, to address this generalized streaming distributed workflows problem class