In the Quest of Trade-off between Job Parallelism and Throughput Adaptive Parameter Tuning of Hadoop

Big data is an emerging concept involving complex data sets which can give new insight and distill new knowledge. In the other hand, Hadoop MapReduce paradigm, a distributed computing software, has been adopted widely in the big data community for large-scale processing. It is known that the implementation of MapReduce with the default configuration results in less number of parallel run- ning job and thus waste of resources in the cluster during MapReduce operation. In fact, poor resource utilization and overall low performance is usually induced by the default configuration during the run-time. This thesis investigated how the cluster resources can be optimally and appropriately utilized during MapReduce operation in order to result in a better job parallelism and throughput. To achieve this, an optimal design is developed which can dynamically change the resources allocation by changing the system level parameters at run-time. The project results showed that the resources are optimally and appropriately utilized during job execution which resulted in better job parallelism and throughput