On the Reusability of Open Test Collections

Creating test collections for modern search tasks is increas-ingly more challenging due to the growing scale and dy-namic nature of content, and need for richer contextualiza-tion of the statements of request. To address these issues, the TREC Contextual Suggestion Track explored an open test collection, where participants were allowed to submit any web page as a result for a personalized venue recom-mendation task. This prompts the question on the reus-ability of the resulting test collection: How does the open nature affect the pooling process? Can participants reliably evaluate variant runs with the resulting qrels? Can other teams evaluate new runs reliably? In short, does the set of pooled and judged documents effectively produce a post hoc test collection? Our main findings are the following: First, while there is a strongly significant rank correlation, the ef-fect of pooling is notable and results in underestimation of performance, implying the evaluation of non-pooled systems should be done with great care. Second, we extensively ana-lyze impacts of open corpus on the fraction of judged docu-ments, explaining how low recall affects the reusability, and how the personalization and low pooling depth aggravate that problem. Third, we outline a potential solution by de-riving a fixed corpus from open web submissions