Recurrent convolutional neural networks for scene labeling

The goal of the scene labeling task is to assign a class label to each pixel in an image. To ensure a good visual coherence and a high class accu-racy, it is essential for a model to capture long range (pixel) label dependencies in images. In a feed-forward architecture, this can be achieved simply by considering a sufficiently large input context patch, around each pixel to be labeled. We propose an approach that consists of a re-current convolutional neural network which al-lows us to consider a large input context while limiting the capacity of the model. Contrary to most standard approaches, our method does not rely on any segmentation technique nor any task-specific features. The system is trained in an end-to-end manner over raw pixels, and mod-els complex spatial dependencies with low infer-ence cost. As the context size increases with the built-in recurrence, the system identifies and cor-rects its own errors. Our approach yields state-of-the-art performance on both the Stanford Back-ground Dataset and the SIFT Flow Dataset, while remaining very fast at test time. 1