Efficient Performance Optimization on Yarn-Based MapReduce Hadoop Framework

Apache Hadoop exposes 180+ configurationparameters for all types of applications and clusters,10-20% of which has a great impact on performanceand efficiency of the execution. The optimalconfiguration settings for one application may not besuitable for another one leading to poor systemresources utilization and long application completiontime. Further, optimizing many parameters is a timeconsuming and a challenging job becauseconfiguration parameters and search space are huge,and users require good knowledge of Hadoopframework. The issue is that the user should adjust atleast the important parameters, e.g. the number ofmap tasks that can run in parallel for a givenapplication. This paper introduces the parameteroptimization algorithm to the key application levelparameter based on input data size and dynamicresource capabilities at any given time for a givenapplication to improve execution time and resourceutilization with nearly zero optimization overhead