Supervised Models for Multimodal Image Retrieval based on Visual, Semantic and Geographic Information

Nowadays, large-scale networked social media need bet-ter search technologies to achieve suitable performance. Multimodal approaches are promising technologies to im-prove image ranking. This is particularly true when meta-data are not completely reliable, which is a rather com-mon case as far as user annotation, time and location are concerned. In this paper, we propose to properly combine visual information with additional multi-faceted informa-tion, to define a novel multimodal similarity measure. More specifically, we combine visual features, which strongly re-late to the image content, with semantic information repre-sented by manually annotated concepts, and geo tagging, very often available in the form of object/subject location. Furthermore, we propose a supervised machine learning approach, based on Support Vector Machines (SVMs), to automatically learn optimized weights to combine the above features. The resulting models is used as a ranking function to sort the results of a multimodal query. 1