In defence of soft-assignment coding

In object recognition, soft-assignment coding enjoys computational efficiency and conceptual simplicity. How-ever, its classification performance is inferior to the newly developed sparse or local coding schemes. It would be highly desirable if its classification performance could be-come comparable to the state-of-the-art, leading to a coding scheme which perfectly combines computational efficiency and classification performance. To achieve this, we revisit soft-assignment coding from two key aspects: classification performance and probabilistic interpretation. For the first aspect, we argue that the inferiority of soft-assignment cod-ing is due to its neglect of the underlying manifold structure of local features. To remedy this, we propose a simple mod-ification to localize the soft-assignment coding, which sur-prisingly achieves comparable or even better performance than existing sparse or local coding schemes while main-taining its computational advantage. For the second as-pect, based on our probabilistic interpretation of the soft-assignment coding, we give a probabilistic explanation to the magic max-pooling operation, which has successfully been used by sparse or local coding schemes but still poorly understood. This probability explanation motivates us to develop a new mix-order max-pooling operation which fur-ther improves the classification performance of the pro-posed coding scheme. As experimentally demonstrated, the localized soft-assignment coding achieves the state-of-the-art classification performance with the highest computa-tional efficiency among the existing coding schemes. 1