Adaptive duplicate detection using learnable string similarity measures

The problem of identifying approximately duplicate records in da-tabases is an essential step for data cleaning and data integration processes. Most existing approaches have relied on generic or man-ually tuned distance metrics for estimating the similarity of poten-tial duplicates. In this paper, we present a framework for improving duplicate detection using trainable measures of textual similarity. We propose to employ learnable text distance functions for each database field, and show that such measures are capable of adapt-ing to the specific notion of similarity that is appropriate for the field’s domain. We present two learnable text similarity measures suitable for this task: an extended variant of learnable string edit distance, and a novel vector-space based measure that employs a Support Vector Machine (SVM) for training. Experimental results on a range of datasets show that our framework can improve dupli-cate detection accuracy over traditional techniques