The performance of mapreduce: an in-depth study

MapReduce-based systems have been widely used for large-scale data analysis. Although these systems achieve storage-system independence, high scalability, and fine-grained fault tolerance, their performance are not satisfactory. According to a recent study [14], MapReduce-based systems are signif-icantly slower than Parallel Database systems in performing a variety of analytic tasks. Some attribute the performance gap between MapReduce-based and Parallel Database sys-tems to architectural design. This speculation yields an in-teresting question: Must a system sacrifice performance to achieve flexibility and scalability? Inspired by the question above, we conducted an in-depth performance study of MapReduce in its open source im-plementation, Hadoop. We identify four factors that have significant performance effect on MapReduce, and investi-gate alternative strategies for each factor. Finally, we eval-uate the performance of MapReduce on a representative yet tractable combinations of the four factors using the same benchmark in [14]. The results show that with proper im-plementation, the performance of MapReduce can be im-proved by a factor of 2.5 to 3.5 and approaches to Parallel Databases. Our results show that it is possible to build a MapReduce-based system that is not only flexible and scal-able, but is also efficient. 1