Effective collection construction for information retrieval evaluation and optimization

The availability of test collections in Cranfield paradigm has significantly benefited the development of models, methods and tools in information retrieval. Such test collections typically consist of a set of topics, a document collection and a set of relevance assessments. Constructing such test collections requires effort of various perspectives such as topic selection, document selection, relevance assessment, and relevance label aggregation etc. The work in the thesis provides a fundamental way of constructing and utilizing test collections in information retrieval in an effective, efficient and reliable manner. To that end, we have focused on four aspects around test collections for information retrieval: (1) Document selection with the objective of identifying relevant documents; this addresses an important issue of the pooling method. (2) Stopping document selection when both the gain of identifying relevant documents and the cost of assessing documents are considered as the optimization goals; this is an important but understudied problem in the construction of test collections. (3) Denoising relevance assessments by aggregating from multiple crowd annotation sources to obtain high-quality relevance assessments; this helps to boost the quality of relevance assessments acquired in a crowdsourcing manner. (4) Optimizing the configuration of retrieval systems on the basis of existing test collections and effectiveness measures. Based on the work in the thesis, we have contributed new models and methods to tackle the issues in test collection construction and made the codes and datasets publicly available to the research community