Towards Large Scale Dense Semantic 3D Reconstruction

With the increasing digitisation of various industries requiring digital twins for virtual interactions or simulations, the need for methods that generate 3D models is more pressing than ever. In particular, 3D reconstruction from images can play a significant part in the solution. When performed together with semantic segmentation, we obtain 3D models that display information about the nature of the scene components, which is valuable for applications in mixed reality and robotics. Moreover, the combined effects of 3D and semantics contribute to improving their quality by enabling reconstruction in areas poorly supported by data through shape priors. One state-of-the-art method considers the problem as a voxel labelling one, which can be solved using convex optimisation. Spectacular results are obtained with high-quality geometry per semantic category, at the cost of limited performances in terms of memory consumption. Hence, state-of-the-art methods are poorly scalable, limiting the size and resolution of the scenes they process as well as the number of semantic categories they can consider. We present several contributions to tackle this challenge. The first consists of a novel data structure dividing the scene into voxel blocks which allow for the presence of a subset of all the semantic categories. However, it necessitates hand-crafted shape priors, and designing them becomes intractable as more categories come into play. We thus introduce a lightweight neural network based on variational inference theory, which takes charge of learning these priors. We then extend it to an octree-based sparse volumetric scene representation bridging the gap between semantic and spatial scalability problems. Moreover, we propose a sensor-adaptive module, which allows our network to process scenes scanned from data degraded by heterogeneous noise models. We conclude with preliminary work on a new method that encodes the scene into latent orthographic space that can be built incrementally from aerial views and decoded into a semantic 3D scene. All along, experiments on indoor, outdoor, real and synthetic datasets with up to 40 categories show that our contributions achieved significant improvements in terms of memory footprint and producing accurate reconstructions without requiring large amounts of training data