S. Marchand–Maillet

Since the early 1990s, content–based visual information retrieval has been an important research topic in computer vision. A large number of systems have been developed as research prototypes, as well as commercial and open source systems. However, still no general breakthrough in perfor-mance has emerged and important real–world applications stay rare. The large amount of available multimedia infor-mation creates a need to develop new tools to explore and retrieve within mixed media databases. The replacement of analog films by digital cameras and the increasing digitisa-tion in fields such as medicine will still increase this need. One of the reasons for the impossibility to show an in-crease in performance is the fact that there is no standard for evaluating the performance of content-based retrieval systems. In the last years a rising number of proposals have been made on how to evaluate or not evaluate the per-formance of visual information retrieval systems which un-derlines the importance of the issue. Several benchmark-ing events such as the Benchathlon, TRECVID and Image-CLEF have been started, with varying success. This article describes work carried out by the University of Geneva on benchmarking visual information retrieval systems. A spe-cial emphasis will be on the Benchathlon and ImageCLEF evaluation events, their methodologies and outcomes.