MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs

MapReduce based data-intensive computing solutions are increas-ingly deployed as production systems. Unlike Internet companies who invent and adopt the technology from the very beginning, tra-ditional enterprises demand easy-to-use software due to the limited capabilities of administrators. Automatic job optimization software for MapReduce is a promising technique to satisfy such require-ments. In this paper, we introduce a toolkit from IBM, called MR-Tuner, to enable holistic optimization for MapReduce jobs. In par-ticular, we propose a novel Producer-Transporter-Consumer (PTC) model, which characterizes the tradeoffs in the parallel execution among tasks. We also carefully investigate the complicated rela-tions among about twenty parameters, which have significant im-pact on the job performance. We design an efficient search algo-rithm to find the optimal execution plan. Finally, we conduct a thorough experimental evaluation on two different types of clusters using the HiBench suite which covers various Hadoop workloads from GB to TB size levels. The results show that the search latency of MRTuner is a few orders of magnitude faster than that of the state-of-the-art cost-based optimizer, and the effectiveness of the optimized execution plan is also significantly improved. 1