Combining object recognition and tracking for augmented reality

UnrestrictedThis thesis takes on the problem of object recognition and tracking specifically applied to augmented reality (AR). The ability to recognize a complex three-dimensional object and compute its pose with respect to the camera opens the door to a vast array of potential AR applications. An object's identity provides a context for selecting virtual content. Its position and orientation provide the necessary geometry to accurately align that content with an image of real world.; General object recognition and tracking are largely unsolved problems in computer vision. The difficulty stems from the many variables that affect appearance, including scale, orientation, illumination, and occlusion. In recent years some approaches, like local feature matching, have shown promising results but ultimately fail to reliably recognize objects in general.; The goal of the present work is to improve on the state of the art in object recognition and tracking in a manner applicable to augmented reality. The ideal system will learn the appearance of a set of objects and then, at runtime, determine which objects are visible and how they are positioned with respect to the camera. Existing solutions generally limit the scope of recognizable objects or depend on additional tracking hardware.; The work demonstrated takes significant strides toward achieving this goal. First addressing the simpler problem of recognizing and tracking planar objects, a novel system of incremental keypoint matching is shown to be both reliable and efficient.; I next address the more general case of nonplanar objects using a dynamic programming algorithm to optimize structure from motion computations. A video sequence can then be processed to model the appearance and geometry of an object, allowing it to be recognized and tracked in future videos.; While all of the techniques mentioned thus far make use of keypoint features, I also demonstrate a means of incorporating edge features. The resulting model, consisting of a collection of 3D points and 3D line segments can be used to improve the accuracy of camera pose estimation and likewise the final AR output