Recurrent convolutional neural networks for scene labeling

Scene labeling is a technique that consist on giving a label to every pixel in an im-age according to the class they belong to. To ensure a good visual coherence and a high class accuracy, it is essential for a scene parser to capture long range de-pendencies on images. In a feed-forward architecture, this can be simply achieved by considering a sufficiently large input context patch, around each pixel to be labeled. We propose an approach consisting of a recurrent convolutional neural network which allows us to consider a large input context, while limiting the ca-pacity of the model. Contrary to most standard approaches, our method does not rely on any segmentation methods, nor any task-specific features. The system is trained in an end-to-end manner over raw pixels, and models complex spatial de-pendencies with low inference cost. As the context size increases with the built-in recurrence, the system identifies and corrects its own errors. Our approach yields state-of-the-art performance on both the Stanford Background Dataset and the SIFT Flow Dataset, while remaining very fast at test time.