Noname manuscript No. (will be inserted by the editor) The Strange Case of Reproducibility vs. Representativeness in Contextual Suggestion Test Collections

the date of receipt and acceptance should be inserted later Abstract The most common approach to measuring the effectiveness of Information Re-trieval systems is by using test collections. The Contextual Suggestion (CS) TREC track provides an evaluation framework for systems that recommend items to users given their geographical context. The specific nature of this track allows the participating teams to identify candidate documents either from the Open Web or from the ClueWeb12 collec-tion, a static version of the web. In the judging pool, the documents from the Open Web and ClueWeb12 collection are distinguished. Hence, each system submission should be based only on one resource, either Open Web (identified by URLs) or ClueWeb12 (identified by ids). To achieve reproducibility, ranking web pages from ClueWeb12 should be the preferred method for scientific evaluation of contextual suggestion systems, but it has been found that the systems that build their suggestion algorithms on top of input taken from the Open Web achieve consistently a higher effectiveness. Because most of the systems take a rather sim-ilar approach to making contextual suggestions, this raises the question whether systems built by researchers on top of ClueWeb12 are still representative of those that would wor