Mesh Based Semantic Modelling for Indoor and Outdoor Scenes

Semantic reconstruction of a scene is important for a va-riety of applications such as 3D modelling, object recog-nition and autonomous robotic navigation. However, most object labelling methods work in the image domain and fail to capture the information present in 3D space. In this work we propose a principled way to generate object labelling in 3D. Our method builds a triangulated meshed representa-tion of the scene from multiple depth estimates. We then de-fine a CRF over this mesh, which is able to capture the con-sistency of geometric properties of the objects present in the scene. In this framework, we are able to generate object hy-potheses by combining information from multiple sources: geometric properties (from the 3D mesh), and appearance properties (from images). We demonstrate the robustness of our framework in both indoor and outdoor scenes. For in-door scenes we created an augmented version of the NYU indoor scene dataset (RGB-D images) with object labelled meshes for training and evaluation. For outdoor scenes, we created ground truth object labellings for the KITTI odom-etry dataset (stereo image sequence). We observe a signifi-cant speed-up in the inference stage by performing labelling on the mesh, and additionally achieve higher accuracies. 1