Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution

The use of crowdsourcing platforms like Amazon Mechan-ical Turk for evaluating the relevance of search results has become an effective strategy that yields results quickly and inexpensively. One approach to ensure quality of worker judgments is to include an initial training period and sub-sequent sporadic insertion of predefined gold standard data (training data). Workers are notified or rejected when they err on the training data, and trust and quality ratings are ad-justed accordingly. In this paper, we assess how this type of dynamic learning environment can affect the workers ’ results in a search relevance evaluation task completed on Amazon Mechanical Turk. Specifically, we show how the distribu-tion of training set answers impacts training of workers and aggregate quality of worker results. We conclude that in a relevance categorization task, a uniform distribution of la-bels across training data labels produces optimal peaks in 1) individual worker precision and 2) majority voting aggregate result accuracy