Crowdsourcing for Language Resources and Evaluation

Crowdsourcing is an efficient approach for knowledge acquisition and data annotation that enables building impressive human-computer systems. In this tutorial we will discuss the relations between Crowdsourcing and Natural Language Processing, focusing on its practical use for Language Resource construction and evaluation. We will describe the established genres of crowdsourcing, show their strengths and weaknesses on real-world examples and case studies, and provide recommendations for ensuring the high quality of the crowdsourced annotation.These materials are published under a CC BY-NC-SA license. Please feel welcome to share them! For viewer convenience, the slides published on Zenodo do not include interactive step-by-step examples