Language Understanding in the Wild: Combining Crowdsourcing and Machine Learning

Social media has led to the democratisation of opinion shar-ing. A wealth of information about public opinions, cur-rent events, and authors insights into specific topics can be gained by understanding the text written by users. How-ever, there is a wide variation in the language used by different authors in different contexts on the web. This diversity in language makes interpretation an extremely challenging task. Crowdsourcing presents an opportunity to interpret the sentiment, or topic, of free-Text. However, the subjec-tivity and bias of human interpreters raise challenges in in-ferring the semantics expressed by the text. To overcome this problem, we present a novel Bayesian approach to lan-guage understanding that relies on aggregated crowdsourced judgements. Our model encodes the relationships between labels and text features in documents, such as tweets, web articles, and blog posts, accounting for the varying reliability of human labellers. It allows inference of annotations that scales to arbitrarily large pools of documents. Our evalu-ation using two challenging crowdsourcing datasets shows that by efficiently exploiting language models learnt from aggregated crowdsourced labels, we can provide up to 25% improved classifications when only a small portion, less than 4% of documents has been labelled. Compared to the six state-of-The-Art methods, we reduce by up to 67% the num-ber of crowd responses required to achieve comparable accu-racy. Our method was a joint winner of the CrowdFlower-CrowdScale 2013 Shared Task challenge at the conference on Human Computation and Crowdsourcing (HCOMP 2013)