saarland.de

In this paper we describe a novel approach for jointly model-ing the text and the visual components of multimedia docu-ments for the purpose of information retrieval(IR). We pro-pose a novel framework where individual components are de-veloped to model different relationships between documents and queries and then combined into a joint retrieval frame-work. In the state-of-the-art systems, a late combination between two independent systems, one analyzing just the text part of such documents, and the other analyzing the visual part without leveraging any knowledge acquired in the text processing, is the norm. Such systems rarely ex-ceed the performance of any single modality (i.e. text or video) in information retrieval tasks. Our experiments indi-cate that allowing a rich interaction between the modalitie