Evaluating Multi-Query Sessions

The standard system-based evaluation paradigm has focused on assessing the performance of retrieval systems in serving the best results for a single query. Real users, however, often begin an interaction with a search engine with a sufficiently under-specified query that they will need to reformulate be-fore they find what they are looking for. In this work we con-sider the problem of evaluating retrieval systems over test collections of multi-query sessions. We propose two families of measures: a model-free family that makes no assumption about the user’s behavior over a session, and a model-based family with a simple model of user interactions over the ses-sion. In both cases we generalize traditional evaluation met-rics such as average precision to multi-query session evalua-tion. We demonstrate the behavior of the proposed metrics by using the new TREC 2010 Session track collection and simulations over the TREC-9 Query track collection