Duplicate discovery on 2 billion internet images

Duplicate image discovery, or discovering duplicate im-age clusters, is a challenging problem for billions of Inter-net images due to the lack of good distance metric which both covers the large variation within a duplicate image cluster and eliminates false alarms. After carefully inves-tigating existing local and global features that have been widely used for large-scale image search and indexing, we propose a two-step approach that combines both local and global features: global descriptors are used to discover seed clusters with high precision, whereas local descriptors are used to grow the seeds to cover good recall. Using effi-cient hashing techniques for both features and the MapRe-duce framework, our system is able to discover about 553.8 million duplicate images from 2 billion Internet images within 13 hours on a 2,000 core cluster. 1