Do we really have to consider covariance matrices for image features?

AbstractMany studies have been made in the past for optimization using covariance matrices of feature points. We first describe how tocompute the covariance matrix of a feature point from the gray levels by integrating existing methods. Then, we experimentallyexamine if thus computed covariance matrices really reflect the accuracy of the feature points. To test this, we do subpixel tem-plate matching and compute the homography and the fundamental matrix. Our conclusion is rather surprising, pointing out impor-tant elements often overlooked. 1. Introduction Detecting feature points is a first step in many vision appli-cations such as 3-D reconstruction and image mosaicing. Since the detected feature points have some uncertainty,many proposed algorithms for geometric computation involve optimization based on the covariance matrices of thefeature points [3, 8, 10, 11]. And various techniques have been proposed for evaluating the covariance matrix from thegray levels [1, 11, 14, 15]. Here, we raise two questions