Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs

Recent trends in semantic image segmentation have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning. In this work, we are interested in understanding the roles of these different tasks in aiding semantic segmentation. Towards this goal, we “plug-in ” human subjects for each of the various components in a state-of-the-art conditional ran-dom field model (CRF) on the MSRC dataset. Comparisons among various hybrid human-machine CRFs give us indi-cations of how much “head room ” there is to improve seg-mentation by focusing research efforts on each of the tasks. One of the interesting findings from our slew of studies was that human classification of isolated super-pixels, while be-ing worse than current machine classifiers, provides a sig-nificant boost in performance when plugged into the CRF! Fascinated by this finding, we conducted in depth analysis of the human generated potentials. This inspired a new ma-chine potential which significantly improves state-of-the-art performance on the MRSC dataset. 1