Learning Spatial Knowledge for Text to 3D Scene Generation

We address the grounding of natural lan-guage to concrete spatial constraints, and inference of implicit pragmatics in 3D en-vironments. We apply our approach to the task of text-to-3D scene generation. We present a representation for common sense spatial knowledge and an approach to ex-tract it from 3D scene data. In text-to-3D scene generation, a user provides as in-put natural language text from which we extract explicit constraints on the objects that should appear in the scene. The main innovation of this work is to show how to augment these explicit constraints with learned spatial knowledge to infer missing objects and likely layouts for the objects in the scene. We demonstrate that spatial knowledge is useful for interpreting natu-ral language and show examples of learned knowledge and generated 3D scenes.