Improving Mapreduce for Incremental Processing Using Map Data Storage

AbstractIn this paper, we propose methods for the improvement of performance of a MapReduce program when it is used for incremental processing. Incremental processing is generally used where data is refreshed periodically to reflect small changes to the input dataset. To reduce the delay in re-computing unchanged data, we introduce methods that selectively compute only data that has been altered. It incorporates the concept of Bloom Filter. Bloom filter is a space-efficient data structure, that can with a certain probability check if the data is modified or not. Traditional systems process the entire data when even a small percentage or none of data is changed. This is time-consuming as well as consumes a huge number of CPU clock cycles additionally to process data that has not been changed. In order to reduce the wastage of CPU clock cycles, a system is proposed wherein a method of execution using Bloom Filter helps improve the performance of the system up to 17% when compared to existing system