Daphne: A Flexible and Hybrid Scheduling Framework in Multi-Tenant Clusters

Distributed computing technologies, as popularized by Hadoop, have been proliferating in Cloud and enterprise computing over ten years, with the capability of processing data across thousands of machines. There is a wide diversity of workloads in such large scale clusters shared by multi-Tenant. Hence, resource utilization and task scheduling become vital to performance and bring challenges to architecture designers. We present Daphne, a hybrid scheduling framework that strikes a tradeoff among three universal scheduling frameworks: 1) centralized scheduling; 2) loose coordination scheduling; and 3) fully distributed scheduling. Daphne defines a matching tree that forwards the application to the fittest scheduler according to the characteristic and priority of the application. Besides, Daphne utilizes resource prediction to increase task throughput significantly while it does not interfere with any running workload. We implement Daphne based on YARN and demonstrate that Daphne improves task throughput by nearly 17%