Comparing Feature Detectors: A bias in the repeatability criteria, and how to correct it

Most computer vision application rely on algorithms finding local correspondences between different images. These algo-rithms detect and compare stable local invariant descriptors centered at scale-invariant keypoints. Because of the impor-tance of the problem, new keypoint detectors and descriptors are constantly being proposed, each one claiming to perform better (or to be complementary) to the preceding ones. This raises the question of a fair comparison between very diverse methods. This evaluation has been mainly based on a repeata-bility criterion of the keypoints under a series of image pertur-bations (blur, illumination, noise, rotations, homotheties, ho-mographies, etc). In this paper, we argue that the classic re-peatability criterion is biased towards algorithms producing re-dundant overlapped detections. To compensate this bias, we propose a variant of the repeatability rate taking into account the descriptors overlap. We apply this variant to revisit the popular benchmark by Mikolajczyk et al. [1], on classic and new feature detectors. Experimental evidence shows that the hierarchy of these feature detectors is severely disrupted by the amended comparator.