A Three-Layered Approach to Facade Parsing Anelo Martinovi¢ 1

Abstract. We propose a novel three-layered approach for semantic seg-mentation of building facades. In the first layer, starting from an over-segmentation of a facade, we employ the recently introduced machine learning technique Recursive Neural Networks (RNN) to obtain a proba-bilistic interpretation of each segment. In the middle layer, initial labeling is augmented with the information coming from specialized facade com-ponent detectors. The information is merged using a Markov Random Field defined over the image. In the highest layer, we introduce weak architectural knowledge, which enforces the final reconstruction to be ar-chitecturally plausible and consistent. Rigorous tests performed on two existing datasets of building facades demonstrate that we significantly outperform the current-state of the art, even when using outputs from lower layers of the pipeline. In the end, we show how the output of the highest layer can be used to create a procedural reconstruction.