Large-scale similarity join with edit-distance constraints

Conference Name:19th International Conference on Database Systems for Advanced Applications, DASFAA 2014. Conference Address: Bali, Indonesia. Time:April 21, 2014 - April 24, 2014.In the age of big data, the data quality problem is more severe than ever. As an essential step in data cleaning, similarity join has attracted lots of attentions from the database community. In this work, to address the similarity join problem with edit-distance constraints, we first improve the partition-based join algorithm for small scale data. Then we extend the algorithm based on MapReduce framework for large-scale data. Extensive experiments on both real and simulated datasets demonstrate the efficiency of our algorithms. ? 2014 Springer International Publishing Switzerland