Toward Consistent Evaluation of Relevance Feedback Approaches in Multimedia Retrieval

Abstract. Many different communities have conducted research on the efficacy of relevance feedback in multimedia information systems. Unlike text IR, performance evaluation of multimedia IR systems tends to conform to the accepted standards of the community within which the work is conducted. This leads to idiosyncratic performance evaluations and hampers the ability to compare different techniques fairly. In this paper we discuss some of the shortcomings of existing multimedia IR system performance evaluations. We propose a common framework in which to discuss the differing techniques proposed for relevance feedback and we develop a strategy for fairly comparing the relative performance of the techniques.