Interference and locality-aware task scheduling for MapReduce applications in virtual clusters

MapReduce emerges as an important distributed program-ming paradigm for large-scale applications. RunningMapRe-duce applications in clouds presents an attractive usage model for enterprises. In a virtual MapReduce cluster, the in-terference between virtual machines (VMs) causes perfor-mance degradation of map and reduce tasks and renders existing data locality-aware task scheduling policy, like de-lay scheduling, no longer effective. On the other hand, vir-tualization offers an extra opportunity of data locality for co-hosted VMs. In this paper, we present a task scheduling strategy to mitigate interference and meanwhile preserving task data locality for MapReduce applications. The strategy includes an interference-aware scheduling policy, based on a task performance prediction model, and an adaptive delay scheduling algorithm for data locality improvement. We im-plement the interference and locality-aware (ILA) scheduling strategy in a virtual MapReduce framework. We evaluated its effectiveness and efficiency on a 72-node Xen-based vir-tual cluster. Experimental results with 10 representative CPU and IO-intensive applications show that ILA is able to achieve a speedup of 1.5 to 6.5 times for individual jobs and yield an improvement of up to 1.9 times in system through-put in comparison with four other MapReduce schedulers