Learning the Relative Importance of Objects from Tagged Images for Retrieval and Cross-Modal Search

Abstract We introduce an approach to image retrieval and auto-tagging that leverages the implicit information about object importance conveyed by the list of keyword tags a person supplies for an image. We propose an unsupervised learning procedure based on Kernel Canonical Correlation Analysis that discovers the relationship between how hu-mans tag images (e.g., the order in which words are men-tioned) and the relative importance of objects and their layout in the scene. Using this discovered connection, we show how to boost accuracy for novel queries, such that the search results better preserve the aspects a human may find most worth mentioning. We evaluate our approach on three datasets using either keyword tags or natural language de-scriptions, and quantify results with both ground truth pa-rameters as well as direct tests with human subjects. Our results show clear improvements over approaches that ei-ther rely on image features alone, or that use words and im-age features but ignore the implied importance cues. Over-all, our work provides a novel way to incorporate high-level human perception of scenes into visual representations for enhanced image search