Context from within: Hierarchical context modeling for semantic segmentation

Conditional Random Fields (CRFs) have been widely adopted in conjunction with Fully Convolutional Networks (FCNs) to model and integrate contextual information in the semantic segmentation procedure. In contrast to existing approaches applying CRFs in parallel or in cascade with FCNs, we propose a new paradigm to incorporate CRFs deeper inside the architecture of FCNs to model the context exhibited within the middle layers of an FCN. We approximate the mean-field inference process of a dense CRF as a multi-dimensional Gated Recurrent Unit (GRU) layer, termed CRF-GRU layer, effectively extracting intermediate context within an FCN. More importantly, multiple CRF-GRU layers can be injected into an FCN to model hierarchical contexts presented in multiple middle layers, showing competitive results on the PASCAL VOC 2012 and PASCAL-Context datasets. Secondly, we contribute a new approach to automatically learn, from the training data, the optimal segmentation architecture of the FCN with multiple CRF-GRU layers injected. The proposed approach relies on Genetic Evolution Strategies to allow the existing architecture to iteratively evolve towards higher accuracy instances. The discovered network not only outperforms state-of-the-art segmentation techniques, but also provides exciting new insights into the design of the segmentation networks.