On Many-Task Big Data Processing: from GPUs to Clouds

Big data processing relies today on complex middleware stacks, comprised of high-level languages, programming mod-els, execution engines, and storage engines. Among these, the execution engine is often built in-house, is tightly inte-grated or even monolithic, and makes coarse assumptions about the hardware. This limits how the future big data processing systems will be able to respond to technologi-cal advances and changes in user workloads. To advance the state-of-the-art, we propose a vision for an open big data execution engine. We identify four key requirements and design a modular, task-based architecture to address them. First, to address performance requirements, we envi-sion the use of multi-layer parallelism, which enables the effi-cient use of heterogeneous platforms (from GPUs to clouds). Second, to address elasticity, we propose the use of online provisioning and allocation of cloud-based resources, which enables economically feasible processing of many big data tasks. Third, to address predictability, we define a perfor-mance envelope, which enables early decisions for selecting and sizing execution platforms. Fourth, we characterize the interaction between our execution engine architecture and the other layers, to enable the efficient operation of the en-tire big data processing stack. We further identify and an-alyze eleven challenges in realizing our vision. Ultimately, these challenges will inspire and motivate a new generation of execution engines for big data processing