Riding the elephant: managing ensembles with hadoop

Many important scientific applications do not fit the tradi-tional model of a monolithic simulation running on thou-sands of nodes. Scientific workflows – such as the Materi-als Genome project, Energy Frontiers Research Center for Gas Separations Relevant to Clean Energy Technologies, cli-mate simulations, and Uncertainty Quantification in fluid and solid dynamics – all run large numbers of parallel anal-yses, which we call scientific ensembles. These scientific ensembles have a large number of tasks with control and data dependencies. Current tools for creating and manag-ing these ensembles in HPC environments are limited and difficult to use; this is proving to be a limiting factor to run-ning scientific ensembles at the large scale enabled by these HPC environments. MapReduce and its open-source imple-mentation, Hadoop, is an attractive paradigm due to the simplicity of the programming model and intrinsic mecha-nisms for handling scalability and fault-tolerance. In this paper, we evaluate the programmability of MapReduce and Hadoop for scientific workflow ensembles