A Comparison of Join Algorithms for Log Processing in

The MapReduce framework is increasingly being used to analyze large volumes of data. One important type of data analysis done with MapReduce is log processing, in which a click-stream or an event log is filtered, aggregated, or mined for patterns. As part of this analysis, the log of-ten needs to be joined with reference data such as informa-tion about users. Although there have been many stud-ies examining join algorithms in parallel and distributed DBMSs, the MapReduce framework is cumbersome for joins. MapReduce programmers often use simple but inefficient al-gorithms to perform joins. In this paper, we describe cru-cial implementation details of a number of well-known join strategies in MapReduce, and present a comprehensive ex-perimental comparison of these join techniques on a 100-node Hadoop cluster. Our results provide insights that are unique to the MapReduce platform and offer guidance on when to use a particular join algorithm on this new plat-form