Mid-level Deep Pattern Mining∗

Mid-level visual element discovery aims to find clusters of image patches that are both representative and discrimi-native. In this work, we study this problem from the prospec-tive of pattern mining while relying on the recently popular-ized Convolutional Neural Networks (CNNs). Specifically, we find that for an image patch, activation extracted from the first fully-connected layer of a CNN have two appealing properties which enable its seamless integration with pat-tern mining. Patterns are then discovered from a large num-ber of CNN activations of image patches through the well-known association rule mining. When we retrieve and visu-alize image patches with the same pattern (See Fig. 1), sur-prisingly, they are not only visually similar but also seman-tically consistent. We apply our approach to scene and ob-ject classification tasks, and demonstrate that our approach outperforms all previous works on mid-level visual element discovery by a sizeable margin with far fewer elements be-ing used. Our approach also outperforms or matches recent works using CNN for these tasks. Source code of the com-plete system is available online. 1