Big Data Processing on Arbitrarily Distributed Dataset

Over the past years, frameworks such as MapReduce and Spark have been introduced to ease the task of developing big data programsand applications. These frameworks significantly reduce the complexity of developing big data programs and applications. However, inreality, many real-world scenarios require pipelining and integration of multiple big data jobs. As the big data pipelines and applicationsbecome more and more complicated, it is almost impossible to manually optimize the performance for each component not to mentionthe whole pipeline/application. At the same time, there are also increasing requirements to facilitate interaction, composition andintegration for big data analytics applications in continuously evolving, integrating and delivering scenarios. In addition, with theemergence and development of cloud computing, mobile computing and the Internet of Things, data are increasingly collected andstored in highly distributed infrastructures (e.g. across data centres, clusters, racks and nodes).To deal with the challenges above and fill the gap in existing big data processing frameworks, we present the Hierarchically DistributedData Matrix (HDM) along with the system implementation to support the writing and execution of composable and integrable big dataapplications. HDM is a light-weight, functional and strongly-typed meta-data abstraction which contains complete information (such asdata format, locations, dependencies and functions between input and output) to support parallel execution of data-driven applications.Exploiting the functional nature of HDM enables deployed applications of HDM to be natively integrable and reusable by other programsand applications. In addition, by analysing the execution graph and functional semantics of HDMs, multiple automated optimizations areprovided to improve the execution performance of HDM data flows. Moreover, by extending the kernel of HDM, we propose a multi-cluster solution which enables HDM to support large scale data analytics among multi-cluster scenarios. Drawing on the comprehensiveinformation maintained by HDM graphs, the runtime execution engine of HDM is also able to provide provenance and historymanagement for submitted applications. We conduct comprehensive experiments to evaluate our solution compared with the currentstate-of-the-art big data processing framework --- Apache Spark