Image-Based Localization

Humans have five senses: sight, sound, smell, taste, and touch. Out of the five, sight is the most important for localizing ourselves or objects in our environment. One important goal of computer vision is to imitate, if not surpass, the human ability to localize oneself based on visual information. Methods targeting this goal are referred to as image-based localization. Existing image-based localization methods can be broadly divided into two categories; retrieval-based and structure-based. While retrieval is suitable for large-scale positioning, structure-based approaches yield accurate 6-DoF poses. These two approaches offer complementary benefits and can be applied sequentially if required. This thesis contains contributions towards both paradigms with a focus on retrieval-based localization. The accuracy of retrieval-based localization depends on the features used to describe the images, the reference selection, and the matching strategy. Recent developments in deep learning have brought forward powerful global feature representations. However, using a feature representation learned for another computer vision task (e.g., image classification) inside a localization framework yields sub-optimal results. This thesis introduces several strategies and techniques to learn feature representations tailored explicitly to the localization task. As a first step towards better localization features, we guide the learning process towards learning a representation where feature distances and geometric distances between images are directly proportional. Knowing the geometric distance between a query and a reference image is highly desirable for accurate localization, especially when the reference images are sparse. As a second step towards better localization features, we address the problem of learning condition invariant features. The ideal feature for retrieval-based localization is invariant to dynamic objects (e.g., cars) and seasonal and daytime changes. Such invariances are learnable by training on a large dataset. To increase large-scale training efficiency, we introduce a new loss function and a new mining strategy. As a final step towards better localization features, we introduce a soft contrastive learning strategy. Unlike existing methods, which divide images into positive and negative classes based on their relative distances, our soft assignment imposes a gradual distinction between close and far images in geometric and feature space. The problem of localization reference selection and image sequence matching strategy often arises as part of a navigation process. Therefore, we are interested in strategies that meet visual navigation requirements. One contribution of this thesis is to formulate a set of navigation requirements for reference selection and sequence matching. Careful reference selection becomes especially important for large-scale localization. The aforementioned feature learning strategies, reference selection, and sequence matching for image-based localization all follow the paradigm of retrieval-based localization. The last part of this thesis introduces an online structure-based localization method for augmented reality applications. In the context of augmented reality, the use of a structure-based method becomes necessary due to the high 6-DoF pose accuracy requirements. To summarize, this thesis contributes to multiple aspects of image-based localization with a focus on large-scale environments