Optimizing Data Analytics Jobs with Fairness Constraints in Datacenter Networks

Thanks to the exponential growth of data that needs to be processed in cloud datacenters, data parallel frameworks, such as MapReduce and Spark, have emerged as foundations of cloud computing. It becomes increasingly significant to improve the performance of data analytics jobs running in a shared cluster. Towards this objective, we investigate important research problems in this dissertation as follows. Utility-Optimal Coflow Scheduling. A coflow represents a set of network flows in the communication stage of a data parallel job. The completion time of a job is determined by the collective behavior of a coflow and influenced by the amount of network bandwidth allocated to it. We focus on the design and implementation of a new utility optimal scheduler across competing coflows, to provide differential treatment to coflows with different degrees of sensitivity, yet still satisfying max-min fairness across these coflows. Job Scheduling across Datacenters. As large volumes of data are generated and stored across geographically distributed datacenters, recent research proposed to distribute the tasks of a data analytics job across datacenters, considering both data locality and network bandwidth. Yet, it remains an open problem in the more general case, where multiple jobs need to fairly share the resources at these datacenters. We study the assignment of tasks belonging to multiple jobs across datacenters, with the specific objective of achieving {\em max-min fairness} across jobs sharing these datacenters, in terms of their job completion times. Multi-Path Routing across Datacenters. To process geo-distributed data, optimizing the network transfer in communication stages also becomes increasingly crucial to application performance, as the inter-datacenter links have much lower bandwidth than intra-datacenter links. We focus on exploiting the flexibility of multi-path routing for inter-datacenter flows of data analytics jobs, with the hope of better utilizing inter-datacenter links and thus improve job performance. We formulate the scheduling problems as lexicographical optimization problems and address the challenges incurred by their multi-objective and discrete nature. We design and implement our solutions as real-world coflow scheduler based on the Varys open-source framework and fair scheduler based on Spark. We also design and implement an optimal multi-path routing and scheduling strategy.Ph.D