Deep vision for indoor understanding and localisation

The challenge considered in this thesis is that of domesticating deep learning-based vision. Firstly, we advance the use of deep learning for indoor architectural comprehension. Secondly, we explore a taming of deep neural networks for low-cost inference such that they can be used in real-time or on constrained hardware. Room or building architecture forms the foundational level of geometry that will be visible within an indoor space, and remains static. Therefore, better understanding of architectural structure enables reasoning over a long period of time without re-training or re-mapping. We introduce Planar R-CNN which can perform unconstrained room layout estimation. Planar R-CNN detects instances of architectural planes, as well as regresses the 3D plane equation for each detection. Planar R-CNN moves room layout estimation beyond strict constraints and can infer arbitrary planar geometries. Following this, we explore visual localisation through architectural reasoning. We introduce LaLaLoc, which learns a latent representation of layout shared between rendered structure and panorama images, enabling cross-modal alignment without explicit layout prediction. LaLaLoc is able to localise across furniture configurations and in previously unvisited scenes given only a floor plan structure as a prior. LaLaLoc++ extends this by unifying representations of layout structure between 2D floor plans and images. Neural network based methods can often be too slow to meet the requirements of a mobile robot, or too large to run on its hardware. We consider low-cost network design, and in particular group convolution as a design tool. Grouped convolution has become increasingly common across the spectrum of speed and accuracy trade-offs architecture design. However, the implications of grouped convolutional layers have remained relatively unexplored. We introduce Mobile R-CNN which, through group convolutional design, performs highly accurate real-time instance segmentation. Finally, we formalise this design of grouped convolutional architectures with the GroSS representation of group convolution blocks, which ultimately enables thorough exploration of the design space where it may not have been feasible before.