RGB-D Camera-Based Parallel Tracking and Meshing Sebastian Lieberknecht∗

Figure 1: A real-time tracking method based on a sparse 3D map is estimating in real-time a consumer RGB-D camera’s motion with respect to an unknown environment (left). At the same time, the method is reconstructing the environment as a dense textured mesh (center). The parallel tracking and meshing opens the way to unprecedented possibilities for several AR applications as the camera motion is accurately estimated and the augmentations are seamlessly integrated in the surrounding environment thanks to convincing occlusions (right). Compared to standard color cameras, RGB-D cameras are designed to additionally provide the depth of imaged pixels which in turn results in a dense colored 3D point cloud representing the envi-ronment from a certain viewpoint. We present a real-time track-ing method that performs motion estimation of a consumer RGB-D camera with respect to an unknown environment while at the same time reconstructing this environment as a dense textured mesh. Unlike parallel tracking and mapping performed with a standard color or grey scale camera, tracking with an RGB-D camera al-lows a correctly scaled camera motion estimation. Therefore, ther