Simple statistical ideas behind efficient sentiment analysis over big data

Social media like Twitter are the primary source of open and big data for statisticians. Although these large data may appear as mainly noise, simple statistical ideas can be applied to extract the meaningful information from this huge repository of words. Seen from the appropriate point of view, those remarkably big data (e.g. dozen of millions of tweets in the case of USA Presidential elections) can be, on the contrary, handled and analyzed in almost real time. After all, if one avoids over modeling of the text data, big data are not that big. We present the experience of Voices from the Blogs (http://voicesformtheblogs.com) which implements some simple statistical ideas of Hopkins and King (2010) to estimate the aggregate opinion of users of social media. These ideas can be summarized in the following steps: i) supervised learning approach (look into the data); ii) no use of ontological dictionaries (best classifier is the human); iii) direct estimation of aggregate opinion rather than post-classification aggregation of individual opinions (let's use Bayes' rule the other way round!) Refs: Hopkins, D.J. and King, G. (2010). A Method of Automated Nonparametric Content Analysis for Social Science. American Journal of Political Science, 54(1): 229-247