Untangling Result List Refinement and Ranking Quality: a Framework for Evaluation and Prediction

Traditional batch evaluation metrics assume that user interaction with search results is limited to scanning down a ranked list. How-ever, modern search interfaces come with additional elements sup-porting result list refinement (RLR) through facets and filters, mak-ing user search behavior increasingly dynamic. We develop an evaluation framework that takes a step beyond the interaction as-sumption of traditional evaluation metrics and allows for batch eval-uation of systems with and without RLR elements. In our frame-work we model user interaction as switching between different sub-lists. This provides a measure of user effort based on the joint effect of user interaction with RLR elements and result quality. We validate our framework by conducting a user study and com-paring model predictions with real user performance. Our model predictions show significant positive correlation with real user ef-fort. Further, in contrast to traditional evaluation metrics, the pre-dictions using our framework, of when users stand to benefit from RLR elements, reflect findings from our user study. Finally, we use the framework to investigate under what condi-tions systems with and without RLR elements are likely to be effec-tive. We simulate varying conditions concerning ranking quality, users, task and interface properties demonstrating a cost-effective way to study whole system performance