The Need for Resilience Research in Coupled Big Compute and Big Data Scientific Applications

Projections and reports about exascale failure modes conclude that we need to protect numerical simula-tions and data analytics from an increasing risk of hardware and software failures and silent data corruptions (SDC). At this scale, hardware and software failures could be as frequent as several per hour. According to [1], the semiconductor industry will have increased difficulty presenting software with an efficient de-pendable hardware layer when feature size will become lower than 10nm (11nm is projected in 2015-2017 according to Intel and NVIDIA). For coupled computation and data analytics at extreme scale, the challenge is to produce correct results in the presence of potentially unreliable hardware and software. After approximately one dozen workshops and reports on exascale resilience, the need for resilience of parallel computations at extreme scale (big compute) is widely accepted. Roadmaps give priority to improv-ing checkpoint restart, developing new programming models and runtimes for resilience, and focusing on detection to limit SDC as much as possible. These roadmaps were primarily concerned with ensuring that large-scale simulations complete and produce correct results. In other contexts such as clouds and grids (big data), many research results concern resilience of workflows, including computation, data storage, and data analytics. However, we are not aware of such roadmaps or research efforts specifically concerning resilience of coupled simulation and data analytics (big compute + big data) in the context of scientific applications o