Robust Wide Baseline Scene Alignment Based on 3D Viewpoint Normalization

Abstract. This paper presents a novel scheme for automatically align-ing two widely separated 3D scenes via the use of viewpoint invariant features. The key idea of the proposed method is following. First, a num-ber of dominant planes are extracted in the SfM 3D point cloud using a novel method integrating RANSAC and MDL to describe the under-lying 3D geometry in urban settings. With respect to the extracted 3D planes, the original camera viewing directions are rectified to form the front-parallel views of the scene. Viewpoint invariant features are ex-tracted on the canonical views to provide a basis for further matching. Compared to the conventional 2D feature detectors (e.g. SIFT, MSER), the resulting features have following advantages: (1) they are very dis-criminative and robust to perspective distortions and viewpoint changes due to exploiting scene structure; (2) the features contain useful local patch information which allow for efficient feature matching. Using the novel viewpoint invariant features, wide-baseline 3D scenes are automat-ically aligned in terms of robust image matching. The performance of the proposed method is comprehensively evaluated in our experiments. It’s demonstrated that 2D image feature matching can be significantly improved by considering 3D scene structure.