Learning Spatial-Semantic Representations from Natural Language Descriptions and Scene Classifications

Abstract — We describe a semantic mapping algorithm that learns human-centric environment models from by interpreting natural language utterances. Underlying the approach is a coupled metric, topological, and semantic representation of the environment that enables the method to infer and fuse information from natural language descriptions with low-level metric and appearance data. We extend earlier work with a novel formulation incorporates spatial layout into a topological representation of the environment. We also describe a factor graph formulation of the semantic properties that encodes human-centric concepts such as type and colloquial name for each mapped region. The algorithm infers these properties by combining the user’s natural language descriptions with image- and laser-based scene classification. We also propose a mechanism to more effectively ground natural language descriptions of spatially non-local regions using semantic cues from other modalities. We describe how the algorithm employs this learned semantic information to propose valid topological hypotheses, leading to more accurate topological and metric maps. We demonstrate that integrating language with other sensor data increases the accuracy of the achieved spatial-semantic representation of the environment. I