Efficient Checkpoint Interval for Speculative Execution in MapReduce

The MapReduce has become popular in big data environment due to its efficient parallel processing. However, MapReduce still has the problem from job delay caused by straggling tasks, which prolong job completion time. In MapReduce framework, although the existing speculative execution mechanism mitigate stragglers, its tasks are slower than their original tasks so this makes job completion time get long when straggling tasks occur. So, in this paper, a checkpoint mechanism is proposed in order to increase the efficiency of speculative execution of MapReduce, and not to prolong job completion time in case of straggling tasks. However, MapReduce produces too much intermediate data; as a result, checkpoint of every intermediate data can still decrease the performance of MapReduce. So, to avoid this problem, the proposed system evaluates checkpoint interval in order to reduce job completion time in case of stragglers. Then, the proposed system defines stragglers using LATE scheduler. The proposed checkpoint interval is based on five parameters: expected job completion time without checkpointing, checkpoint overhead time, rework time, down time and restart time. Experimental results show that the proposed system leads to less completion time, rework time and checkpoint overhead