Stateful bulk processing for incremental analytics

This work addresses the need for stateful dataflow programs that can rapidly sift through huge, evolving data sets. These data-inten-sive applications perform complex multi-step computations over successive generations of data inflows, such as weekly web crawls, daily image/video uploads, log files, and growing social networks. While programmers may simply re-run the entire dataflow when new data arrives, this is grossly inefficient, increasing result la-tency and squandering hardware resources and energy. Alterna-tively, programmers may use prior results to incrementally incor-porate the changes. However, current large-scale data processing tools, such as Map-Reduce or Dryad, limit how programmers in-corporate and use state in data-parallel programs. Straightforward approaches to incorporating state can result in custom, fragile code and disappointing performance. This work presents a generalized architecture for continuous bulk processing (CBP) that raises the level of abstraction for building in-cremental applications. At its core is a flexible, groupwise process-ing operator that takes state as an explicit input. Unifying stateful programming with a data-parallel operator affords several funda-mental opportunities for minimizing the movement of data in the underlying processing system. As case studies, we show how one can use a small set of flexible dataflow primitives to perform web analytics and mine large-scale, evolving graphs in an incremental fashion. Experiments with our prototype using real-world data in-dicate significant data movement and running time reductions rel-ative to current practice. For example, incrementally computing PageRank using CBP can reduce data movement by 46 % and cut running time in half