Tolhit – A Scheduling Algorithm for Hadoop Cluster

AbstractWith the accretion in use of Internet in everything, a prodigious influx of data is being observed. Use of MapReduce as a programming model has become pervasive for processing such wide range of Big Data Applications in cloud computing environment. Apache Hadoop is the most prominent implementation of MapReduce, which is used for processing and analyses of such large scale data intensive applications in a highly scalable and fault tolerant manner. Several scheduling algorithms have been proposed for Hadoop considering various performance goals. In this work, a new scheme is introduced to aid the scheduler in identifying the nodes on which stragglers can be executed. The proposed scheme makes use of resource utilization and network information of cluster nodes in finding the most optimal node for scheduling the speculative copy of a slow task. The performance evaluation of the proposed scheme has been done by series of experiments. From the performance analysis 27% improvement in terms of the overall execution time has been observed over Hadoop Fair Scheduler (HFS)