PStorM: Profile Storage and Matching for Feedback-Based Tuning of MapReduce Jobs Mostafa Ead∗ Amazon

The MapReduce programming model has become widely adopted for large scale analytics on big data. MapReduce systems such as Hadoop have many tuning parameters, many of which have a sig-nificant impact on performance. The map and reduce functions that make up a MapReduce job are developed using arbitrary pro-gramming constructs, which make them black-box in nature and therefore renders it difficult for users and administrators to make good parameter tuning decisions for a submitted MapReduce job. An approach that is gaining popularity is to provide automatic tun-ing decisions for submitted MapReduce jobs based on feedback from previously executed jobs. This approach is adopted, for ex-ample, by the Starfish system. Starfish and similar systems base their tuning decisions on an execution profile of the MapReduce job being tuned. This execution profile contains summary infor-mation about the runtime behavior of the job being tuned, and it is assumed to come from a previous execution of the same job. Managing these execution profiles has not been previously stud-ied. This paper presents PStorM, a profile store and matcher that accurately chooses the relevant profiling information for tuning a submitted MapReduce job from the previously collected profiling information. PStorM can identify accurate tuning profiles even for previously unseen MapReduce jobs. PStorM is currently integrated with the Starfish system, although it can be extended to work with any MapReduce tuning system. Experiments on a large number of MapReduce jobs demonstrate the accuracy and efficiency of pro-file matching. The results of these experiments show that the pro-files returned by PStorM result in tuning decisions that are as good as decisions based on exact profiles collected during pervious ex-ecutions of the tuned jobs. This holds even for previously unseen jobs, which significantly reduces the overhead of feedback-driven profile-based MapReduce tuning