Evaluating Answer Passages using Summarization Measures

Passage-based retrieval models have been studied for some time and have been shown to have some benefits for doc-ument ranking. Finding passages that are not only topi-cally relevant, but are also answers to the users ’ questions would have a significant impact in applications such as mo-bile search. To develop models for answer passage retrieval, we need to have appropriate test collections and evaluation measures. Making annotations at the passage level is, how-ever, expensive and can have poor coverage. In this pa-per, we describe the advantages of document summarization measures for evaluating answer passage retrieval and show that these measures have high correlation with existing mea-sures and human judgments. 1