MRONLINE: MapReduce Online Performance Tuning

MapReduce job parameter tuning is a daunting and time consum-ing task. The parameter configuration space is huge; there are more than 70 parameters that impact job performance. It is also difficult for users to determine suitable values for the parameters without first having a good understanding of the MapReduce application characteristics. Thus, it is a challenge to systematically explore the parameter space and select a near-optimal configuration. Ex-tant offline tuning approaches are slow and inefficient as they entail multiple test runs and significant human effort. To this end, we propose an online performance tuning sys-tem, MRONLINE, that monitors a job’s execution, tunes associ-ated performance-tuning parameters based on collected statistics, and provides fine-grained control over parameter configuration. MRONLINE allows each task to have a different configuration, in-stead of having to use the same configuration for all tasks. More-over, we design a gray-box based smart hill climbing algorithm that can efficiently converge to a near-optimal configuration with high probability. To improve the search quality and increase con-vergence speed, we also incorporate a set of MapReduce-specific tuning rules in MRONLINE. Our results using a real implemen-tation on a representative 19-node cluster show that dynamic per-formance tuning can effectively improve MapReduce application performance by up to 30 % compared to the default configuration used in YARN