Novel Algorithm with In-node Combiner for enhanced performance of MapReduce on Amazon EC2

To distribute large datasets over multiple commodity servers and to perform a parallel computation a Hadoop framework is used. A question that arises with any program is efficiency of the program and its completion time. MapReduce programming model uses the divide and conquer rule, the map (reduce) tasks consists of specific, well defined phases for data processing. However only map and reduce functions are custom and their execution time can be predicted by user. The execution time for the remaining phases is generic and totally depends on the amount of data processed by the phase and the performance of underlying Hadoop cluster. The optimization of I/O can contribute towards the better performance. Hence in this paper, we will look into such I/O bottlenecks that Hadoop framework faces and a possible solution to overcome the same. We have introduced an approach that will help to optimize I/O, the combining at a node level. This design has taken the traditional combiner to a next level wherein the number of intermediate results are reduced with the help of combiner at a node level which results in reduced network traffic between mappers and reducers