Optimizing Big Data Analytics Frameworks in Geographically Distributed Datacenters

A variety of Internet applications rely on big data analytics frameworks to efficiently process large volumes of raw data. As such applications expand to a continental or even global scale, their raw data input can be generated and stored in different datacenters. The performance of data analytics jobs is likely to suffer, as transferring data among workers located in different datacenters is expensive. In this dissertation, we propose a series of system optimizations for big data analytics frameworks that are deployed across geographically distributed datacenters. Our works optimize the following three components in their architectural design: Inter-datacenter network transfers. Few measures have been taken to handle the unpredictability and scarcity of inter-datacenter available bandwidth. With the awareness of job-level performance, we focus on expediting inter-datacenter coflows, which are collections of parallel flows associated with job-level communication requirements. We propose novel scheduling and routing strategies to minimize the coflow completion times. These strategies are implemented in Siphon, a software-defined overlay network deployed atop inter-datacenter networks. Shuffles. Data analytics frameworks all provide a set of parallel operators as application building blocks to simplify development. Some operators trigger all-to-all transfers among worker nodes, known as shuffles. Shuffles are the source of inter-datacenter traffic, and its performance is critical. Our work focuses on improving the network utilization during shuffles, by adopting a new push-based mechanism that allows early start of individual flow transfers. Graph analytics APIs. Graph analytics is among the most important algorithms that are natively supported by typical big data analytics frameworks. Bulk Synchronous Parallel (BSP) is the state-of-the-art synchronization model to parallelize graph algorithms, which generates a significant amount of traffic among worker nodes iteratively. We propose a Hierarchical Synchronous Parallel (HSP) model, which is designed to reduce the demand for inter-datacenter transfers. HSP achieves this goal without sacrificing algorithm correctness, resulting in a better performance and lower monetary cost. We have implemented and tested the prototypes based on Apache Spark, one of the most popular data analytics frameworks. Extensive experimental results on real public clouds across multiple geographical regions have shown their effectiveness.Ph.D