Language modeling for bag-of-visual words image categorization

In this paper, we propose two ways of improving image clas-sification based on bag-of-words representation [25]. Two shortcomings of this representation are the loss of the spa-tial information of visual words and the presence of noisy visual words due to the coarseness of the vocabulary build-ing process. On the one hand, we propose a new representa-tion of images that goes further in the analogy with textual data: visual sentences, that allows us to “read ” visual words in a certain order, as in the case of text. We can there-fore consider simple spatial relations between words. We also present a new image classification scheme that exploits these relations. It is based on the use of language mod-els, a very popular tool from speech and text analysis com-munities. On the other hand, we propose new techniques to eliminate useless words, one based on geometric proper-ties of the keypoints, the other on the use of probabilistic Latent Semantic Analysis (pLSA). Experiments show that our techniques can significantly improve image classification, compared to a classical Support Vector Machine-based clas-sification