Interactive Learning of Spatial Knowledge for Text to 3D Scene Generation

We present an interactive text to 3D scene generation system that learns the expected spatial layout of objects from data. A user provides input natural language text from which we extract explicit constraints on the objects that should appear in the scene. Given these explicit constraints, the sys-tem then uses prior observations of spa-tial arrangements in a database of scenes to infer the most likely layout of the ob-jects in the scene. Through further user interaction, the system gradually adjusts and improves its estimates of where ob-jects should be placed. We present exam-ple generated scenes and user interaction scenarios.