Look and Think Twice: Capturing Top-Down Visual Attention with Feedback Convolutional Neural Networks∗

While feedforward deep convolutional neural networks (CNNs) have been a great success in computer vision, it is important to note that the human visual cortex generally contains more feedback than feedforward connections. In this paper, we will briefly introduce the background of feed-backs in the human visual cortex, which motivates us to de-velop a computational feedback mechanism in deep neural networks. In addition to the feedforward inference in tra-ditional neural networks, a feedback loop is introduced to infer the activation status of hidden layer neurons accord-ing to the “goal ” of the network, e.g., high-level semantic labels. We analogize this mechanism as “Look and Think Twice. ” The feedback networks help better visualize and understand how deep neural networks work, and capture visual attention on expected objects, even in images with cluttered background and multiple objects. Experiments on ImageNet dataset demonstrate its effectiveness in solving tasks such as image classification and object localization. 1