Multimedia semantic indexing using model vectors

In this paper we propose a novel method for multimedia se-mantic indexing using model vectors. Model vectors provide a semantic signature for multimedia documents by capturing the de-tection of concepts broadly across a lexicon using a set of in-dependent binary classifiers. While recent techniques have been developed for detecting simple generic concepts such as indoors, outdoors, nature, manmade, faces, people, speech, music, and so forth [1], these labels directly support only a small number of queries. Model vectors address the problem of answering queries for which relationships to specific concepts is either unknown or indirect by developing a basis across across the lexicon. In the simplest case, each model vector dimension corresponds to the confidence score by which a corresponding concept from the lexi-con is detected. However, we show how other information such as relevance, reliability and concept correlation can also be incorpo-rated. Overall, the model vectors can be used in a variety of meth-ods for multimedia indexing, including model-based retrieval, rel-evance feedback searching and concept querying. In this paper, we present the model vector method and study different strategies for computing and comparing model vectors. We empirically evaluate the retrieval effectiveness of the model vector approach compared to other search methods in a large video retrieval testbed