Performance and Energy Efficiency of Big Data Applications in Cloud Environments: A Hadoop Case Study

The exponential growth of scientic and business data has resulted in theevolution of the cloud computing environments and the MapReduce parallelprogramming model. The focus of cloud computing is increased utilizationand power savings through consolidation while MapReduce enables largescale data analysis. Hadoop, an open source implementation of MapReducehas gained popularity in the last few years. In this paper, we evaluate Hadoopperformance in both the traditional model of collocated data and computeservices as well as consider the impact of separating out the services. Theseparation of data and compute services provides more exibility in environments where data locality might not have a considerable impact such asvirtualized environments and clusters with advanced networks. In this paper,we also conduct an energy eciency evaluation of Hadoop on physical andvirtual clusters in dierent congurations. Our extensive evaluation showsthat: (1) coexisting virtual machines on servers decrease the disk throughput;(2) performance on physical clusters is signicantly better than on virtualclusters; (3) performance degradation due to separation of the services depends on the data to compute ratio; (4) application completion progresscorrelates with the power consumption and power consumption is heavilyapplication specic. Finally, we present a discussion on the implications ofusing cloud environments for big data analyses