Hydra: A Spatial Perception Engine for Constructing and Optimizing 3D Scene Graphs in Real-time

3D scene graphs have emerged as a powerful map representation for robotics. Scene graphs encode spatial and semantic concepts at multiple levels of abstraction as nodes in a graph, and use edges to represent relationships. Such representations offer an efficient way to model diverse environments, and can be used as an aid for planning tasks requiring semantic knowledge. However, current approaches that construct scene graphs lack the ability to operate in real-time on robots. This thesis addresses this research gap, investigating how to build scene graphs from sensor data. We first introduce the concept of a 3D scene graph, and then detail our contributions to Kimera, the first work to build a hierarchical 3D scene graph directly from visual-inertial sensor data in post-processing. We discuss experiments that explore the runtime performance of Kimera, and produce scene graphs for real-life environments. We then introduce Hydra, a real-time spatial perception system that can construct 3D scene graphs incrementally from visual-inertial sensor data, overcoming several limitations of Kimera. We also propose an approach for enhancing traditional vision-based loop closure detection using scene graphs, and detail the first method for correcting a scene graph in response to odometric drift. We provide an extensive evaluation of Hydra, including comparing the produced scene graphs to the scene graphs produced by Kimera. We discuss the suitability of Hydra for usage on mobile robots; towards this we show a scene graph produced by data collected on a real robot and evaluate the runtime performance of Hydra on an embedded processor. Finally, we turn to applications of Hydra to planning and navigation tasks in robotics. We detail how 3D scene graphs can be used as observations in a reinforcement learning framework. Experiments evaluating this framework show that the hierarchy of the scene graph improves the effectiveness of the learned policy. We then discuss the role that Hydra could play in deploying learned policies to real robots.S.M