The Need for Resilience Research in Workflows of Big Compute and Big Data Scientific Applications

Projections and reports about exascale failure modes conclude that we need to protect numerical simula-tions and data analytics from an increasing risk of hardware and software failures and silent data corruptions (SDC) [1, 4]. At this scale, hardware and software failures could be as frequent as ten or more per day. Ac-cording to [9], the semiconductor industry will have increased difficulty presenting software with an efficient dependable hardware layer when feature size will become lower than 10 nm (11 nm is projected in 2015-2017 according to Intel and NVIDIA). For workflows of computation and data analytics at extreme scale, the challenge is to produce correct results in the presence of potentially unreliable hardware and software. After many workshops and reports on exascale resilience [2, 3, 5, 6, 10], the need for resilience of paral-lel computations at extreme scale (big compute) is widely accepted. Roadmaps give priority to improving checkpoint restart, developing new programming models and runtimes for resilience, developing resilient algorithms and focusing on detection to limit SDC as much as possible. These roadmaps were primarily concerned with ensuring that large-scale simulations complete and produce correct results. In other contexts such as clouds and grids (big data), many research results concern resilience of workflows, including com-putation, data storage, and data analytics. However, we are not aware of such roadmaps or research efforts specifically concerning resilience of workflows of simulation and data analytics (big compute + big data) i