Attention to Scale: Scale-aware Semantic Image Segmentation

Incorporating multi-scale features to deep convolutional neural networks (DCNNs) has been a key element to achieve state-of-art performance on semantic image segmentation benchmarks. One way to extract multi-scale features is by feeding several resized input images to a shared deep net-work and then merge the resulting multi-scale features for pixel-wise classification. In this work, we adapt a state-of-art semantic image segmentation model with multi-scale input images. We jointly train the network and an attention model which learns to softly weight the multi-scale features, and show that it outperforms average- or max-pooling over scales. The proposed attention model allows us to diagnos-tically visualize the importance of features at different posi-tions and scales. Moreover, we show that adding extra su-pervision to the output of DCNN for each scale is essential to achieve excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with exhaustive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014. 1