Beyond spatial pyramids: Receptive field learning for pooled image features

We examine the effect of receptive field designs on the classification accuracy in the commonly adopted pipeline of image classification. While existing al-gorithms use manually defined spatial regions for pooling, we argue that learn-ing more adaptive receptive field increases performance even with a significantly smaller codebook size at the coding layer. To this end, we adopt the idea of over-completeness by starting with a large number of receptive field candidates, and train a classifier that only uses a sparse subset of them. An efficient algorithm based on grafting is proposed to perform feature selection from a set of pooling candidates. With this method, we achieve the best published performance on the CIFAR-10 dataset, using a much lower dimensional feature space than previous methods did.