Noname manuscript No. (will be inserted by the editor) Integrating Function, Geometry, Appearance for Scene Parsing

Abstract Visual scene understanding is a fundamen-tal task in computer vision systems. Traditional appear-ance-based classification paradigms struggle to cope with the view and appearance variations of indoor scenes and functional objects. In this paper, we present a Stochas-tic Scene Grammar (SSG) model to parse indoor im-ages. The grammar is defined on a Function-Geometry-Appearance (FGA) hierarchy based on two observa-tions: i) Functionality is the most essential property to define an indoor object, e.g.“an object to sit on” defines a chair, ii) The geometry (3D shape) of an ob-ject is designed to serve its function. We formulate the nature of the object functionality and contextual rela-tions into the Stochastic Scene Grammar model, which characterizes a joint distribution over the FGA hierar-chy. This hierarchical structure includes both functiona