A comparison of evaluation measures given how users perform on search tasks

Abstract Information retrieval has a strong founda-tion of empirical investigation: based on the position of relevant resources in a ranked answer list, a variety of system performance metrics can be calculated. One of the most widely reported measures, mean average precision (MAP), provides a single numerical value that aims to capture the overall performance of a retrieval system. However, recent work has suggested that broad measures such as MAP do not relate to actual user per-formance on a number of search tasks. In this paper, we investigate the relationship between various retrieval metrics, and consider how these reflect user search per-formance. Our results suggest that there are two dis-tinct categories of measures: those that focus on high precision in an answer list, and those that attempt to capture a broader summary, for example by including a recall component. Analysis of runs submitted to the TREC terabyte track in 2006 suggests that the relative performance of systems can differ significantly depend-ing on which group of measures is being used