Learning to combine trained distance metrics for duplicate detection in databases

The problem of identifying approximately duplicate records in da-tabases has previously been studied as record linkage, the merge/pur-ge problem, hardening soft databases, and field matching. Most ex-isting approaches have focused on efficient algorithms for locating potential duplicates rather than precise similarity metrics for com-paring records. In this paper, we present a domain-independent method for improving duplicate detection accuracy using machine learning. First, trainable distance metrics are learned for each field, adapting to the specific notion of similarity that is appropriate for the field’s domain. Second, a classifier is employed that uses sev-eral diverse metrics for each field as distance features and classifies pairs of records as duplicates or non-duplicates. We also propose an extended model of learnable string distance which improves over an existing approach. Experimental results on real and synthetic datasets show that our method outperforms traditional techniques