vRead: Efficient Data Access for Hadoop in Virtualized Clouds

With its unlimited scalability and on-demand access to computation and storage, a virtualized cloud platform is the perfect match for big data systems such as Hadoop. However, virtualization introduces a significant amount of overhead to I/O intensive applications due to device virtualization and VMs or I/O threads scheduling delay. In particular, device virtualization causes significant CPU overhead as I/O data needs to be moved across several protection boundaries. We observe that such overhead especially affects the I/O perfor-mance of the Hadoop distributed file system (HDFS). In fact, data read from an HDFS datanode VM must go through virtual devices multiple times — incurring non-negligible virtualization overhead — even though both client VM and datanode VM may be running on the same machine. In this paper, we propose vRead, a pro-grammable framework which connects I/O flows from HDFS ap-plications directly to their data. vRead enables direct “reads ” to the disk images of datanode VMs from the hypervisor. By doing so, vRead can significantly avoid device virtualization overhead, resulting in improved I/O throughput as well as CPU savings for Hadoop workloads and other applications relying on HDFS