Dynamic Cluster Configuration Algorithm in

Abstract — With the exponential growth of Data in recent time, industry and academia started looking for an intelligent data analysis tool that would satisfy the need of current requirements in storage and analysis of huge amount of data. The data growth is largely due to the impact of social media, scientific experiments and file logs created by different departments around the globe. MapReduce, proposed by Google in 2004 became popular for doing large scale data analysis. Industry is also concentrating on providing resources and services on demand, cost effectively and with high performance. Implementing MapReduce in cloud requires creation of clusters, where the Map and Reduce operations can be performed. Optimizing the overall resource utilization without compromising with the efficiency of availing services is the need for the hour. Selecting right set of nodes to form cluster plays a major role in improving the performance of the cloud. As a huge amount of data transfer takes place during the data analysis phase, network latency becomes the defining factor in improving the QoS of the cloud. In this paper we propose a novel Cluster Configuration algorithm that selects optimal nodes in a dynamic cloud environment to configure a cluster for running MapReduce jobs. The algorithm is cost optimized, adheres to global resource utilization and provides high performance to the clients. The proposed Algorithm gives a performance benefit of 35 % on all reconfiguration based cases and 45 % performance benefit on best cases