Differences in Search Engine Evaluations Between Query Owners and Non-Owners

The query-document relevance judgments used in web search engine evaluation are traditionally provided by human as-sessors who have no particular association with the specific queries selected for the evaluation. Most commonly, queries are randomly sampled from search logs and in turn randomly assigned to the human assessors. In this paper, we consider a very different approach in which we instead ask the hu-man assessors to provide their own queries from their recent search experiences. Using these queries as our sample, we compare the relevance judgments from the “owners ” of the queries to the relevance judgments of the non-owners. We conduct experiments which reveal that query owner-ship has a substantial and beneficial impact on the accu-racy of relevance judgments. In particular, we observe that owners are more consistently able to distinguish a higher quality set of search results from a lower quality set in a blind comparison. The implication for web search evaluation is that query owners provide more valuable relevance judg-ments than non-owners, presumably due to the background knowledge associated with their queries. We quantify the benefit of using owner assessments versus non-owner assess-ments in terms of sample size reduction. We also touch on some of the practical challenges associated with using query owners as assessors