Evaluating interactive cross-language information retrieval: Document selection

Abstract. The problem of nding documents that are written in a lan-guage that the searcher cannot read is perhaps the most challenging application of Cross-Language Information Retrieval (CLIR) technol-ogy. The rst Cross-Language Evaluation Forum (CLEF) provided an excellent venue for assessing the performance of automated CLIR tech-niques, but little is known about how searchers and systems might in-teract to achieve better cross-language search results than automated systems alone can provide. This paper explores the question of how in-teractive approaches to CLIR might be evaluated, suggesting an initial focus on evaluation of interactive document selection. Important evalua-tion issues are identied, the structure of an interactive CLEF evaluation is proposed, and the key research communities that could be brought to-gether by such an evaluation are introduced.