The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval

This paper details the collection, systems and evaluation methods used in the IR Task of the CLEF 2016 eHealth Evaluation Lab. This task investigates the effectiveness of web search engines in providing access to medical information for common people that have no or little medical knowledge. The task aims to foster advances in the development of search technologies for consumer health search by providing resources and evaluation methods to test and validate search systems. The problem considered in this year’s task was to retrieve web pages to support the information needs of health consumers that are faced by a medical condition and that want to seek relevant health information online through a search engine. As part of the evaluation exercise, we gathered 300 queries users posed with respect to 50 search task scenarios. The scenarios were developed from real cases of people seeking health information through posting requests of help on a web forum. The presence of query variations for a single scenario helped us capturing the variable quality at which queries are posed. Queries were created in English and then translated into other languages. A total of 49 runs by 10 different teams were submitted for the English query topics; 2 teams submitted 29 runs for the multilingual topics