Peer-to-Peer Netw. Appl. DOI 10.1007/s12083-013-0213-7 Handling partitioning skew in MapReduce using LEEN

Abstract MapReduce is emerging as a prominent tool for big data processing. Data locality is a key feature in MapRe-duce that is extensively leveraged in data-intensive cloud systems: it avoids network saturation when processing large amounts of data by co-allocating computation and data stor-age, particularly for the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew (Parti-tioning skew refers to the case when a variation in either the intermediate keys ’ frequencies or their distributions or both among different data nodes) huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications severe performance degrada-tion due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. In this paper, we develop a novel algorithm named LEEN for locality-aware and fairness-aware key partition-ing in MapReduce. LEEN embraces an asynchronous ma