A spatial consistent CRF for semantic image segmentation

The goal of semantic image segmentation is to separate an image into parts of different semantic content, i.e. with respect to high level object classes such as "cars" or "persons". A new approach for this task based on a probabilistic graphical model formulation is proposed in this thesis. Key elements of the method are a set of proposal segments. Each proposal segment is generated by existing algorithms to partition an object from the rest of the image. The presented method follows the idea of segmentation by classifying super-pixels (small clusters of neighboring pixels), which are determined by intersecting all proposal segments. A conditional random field (CRF) consisting of a two layer spatial hierarchy is formulated. While the bottom layer represents the class assignments of super-pixels, the top level contains assignments for the proposal segments. This results in two super-pixels being connected by segments that contain both. Many latent random variables are present in the proposed CRF, which renders standard Machine Learning approaches for parameter learning computationally infeasible. Therefore, two alternative learning schemes motivated by the spatial object class distribution in the image, are presented and evaluated in this thesis. The segmentation performance of the proposed method is compared different baseline methods in the state-of-the-art setting on the dataset of the VOC Segmentation Challenge 2011. Based on the experimental results extractions of image information in segments are identified as the component that limit model performance most. Finally, several promising approaches are suggested for future research to overcome current model limitations