Big Data in the Background: Maximizing Productivity while Minimizing Virtual Machine Interference

Despite the use of virtualization to share individual servers, data centers are still often only lightly loaded, leaving large amounts of spare capacity idle. In some ways, Big Data applications are an ideal fit for using this excess capacity to perform meaningful work, yet the high level of interference between interactive and batch processing workloads currently prevents this from be-ing a practical solution in virtualized environments. In this paper we study both the amount of spare capacity that could potentially be used for Big Data processing, and the limitations of current interference isolation tech-niques. Within our university data center we find that an average of 90%, 53%, and 89 % of CPU, memory, and disk IO capacity are left idle; an analysis of Wikipedia’s 137 application servers shows similar results. We then evaluate the impact of colocating a Hadoop VM with an interactive website, and find that median web server re-sponse time can increase by over seven times. Finally, we propose how scheduling mechanisms in both the vir-tualization layer and the Hadoop job tracker could be en-hanced to improve this situation.