Inverse Procedural Modeling of Buildings

3D city modeling is a thriving area of research, as high quality models of real-world cities are in ever-rising demand. These models are used not only among architects and urban planners, but also find their application in navigation, virtual tourism, and entertainment industry. Manual modeling of individual buildings usually provides good results, but the process is very time consuming and expensive. Current automatically-built models using Structure from Motion, followed by simple plane fitting and texturing are a good starting point, but provide inadequate 3D visual perception. No matter the capturing technology, the resulting models are deficient, particularly when dealing with thin objects, fragmented volumes and reflective surfaces. Furthermore, conventional bottom-up models lack any semantic knowledge about the scene. Yet, adding a good understanding of what it is that needs to be modeled is a strong cue, not only to improve the visual and 3D quality of the model, but also to substantially widen its usage. Conversely, procedural modeling provides an effective way to create detailed and realistic 3D building models that do come with all the semantic labels required. This elegant yet powerful framework representsmodels such as buildings through instantiations of a series of parameterized rules, forming a grammar. The resulting procedural models are compact, rich in terms of semantics, and allow for more aesthetic rendering than would be possible from pure 3D capturing. Thus far, procedural modeling has largely been used for synthesizing virtual buildings. In this thesis, we investigate how procedural models can be used in the context of urban reconstruction. Our ultimate goal isto automatically create procedural models of structures as-built, a process referred to as inverse procedural modeling. The main challenge in this process is to determine the appropriate rules of the procedural grammar and their parameters, which typically results in a large search space. In the first part of the thesis we assume the grammar rules are already known, while parameters are allowed to vary. We develop a system for 3D building reconstruction where the grammar leads the modeling process andreceives structural information from object detectors. A drawback of this approach is that the grammar needs to be selected beforehand. Therefore, we develop an algorithm for automatic selection of the appropriate style grammar based on the visual recognition of the architectural style of the observed building. Presently, the main drawback of procedural grammars is that expert architectural knowledge is needed for their creation, which is a non-trivial manual process. Moreover, the abundance of different architectural styles in the world would require many such grammars to bedesigned. To tackle this problem, in the second part of the thesis we simplify the prior knowledge necessary for building reconstruction to a set of general and style-independent architectural principles. We use these weaker priors in a bottom-up approach, by producing high-quality semantic labeling of perspective images and Structure-from-Motion point clouds. This labeling is afterwards transformed into building-specific procedural models, allowing realistic rendering. In the third part of the thesis, we address the problem of procedural grammar scarcity by proposing to learn the grammars from data. First, weshow that probabilistic grammars can be learned from annotated facade imagery. The inferred grammars are shown to be comparable to expert-written grammars on the task of facade reconstruction. Second, weeliminate the need for manual image annotation by replacing it with the previously proposed automatic facade labeling approach. Finally, the learned representations are shown to be useful for virtual facade synthesis, facade comparison and retrieval. The proposed models have been evaluated on several datasets of urban scenes, advancing the state of the art in terms of accuracy and speed. More importantly, it is the conclusion of this thesis that the problem of inverse procedural modeling of buildings could be solved with grammarlearning from labeled and noisy data, obviating the need for a human in the loop, and opening up novel directions for future research.Martinović A., ''Inverse procedural modeling of buildings'', Proefschrift voorgedragen tot het behalen van het doctoraat in de ingenieurswetenschappen, KU Leuven, September 2015, Leuven, Belgium.status: publishe