Analysing the simultaneous effects of personalities, demographics, competition and collaboration : a case study with Indian politics

Twitter is widely regarded as a fast growing social networking medium. The massive amounts of data available on twitter encourage researchers to crawl and analyse data from it. The purpose of this project is to create a complete system which contains a twitter crawler, a multidimensional database, a data digester and an analytics engine. The crawler is used for the crawling of tweets, the multidimensional database for storing the crawled data and a data digester for formatting the data crawled before storing it into the database while the analytics engine used to perform analysis on the data. Due to the magnitude of this project, multiple students are allocated to it. The author of this report was involved in the schema design of the twitter database and came up with a list of the types of content of the tweets data to be crawled by the twitter crawler. The data digester and part-of-speech tagging for the analytics engine were developed by the author of this report as well. The testing of the system is spread over a period of 4 days, from 14th to 17th March 2014. Within these 4 days, a total of 8,021,933 tweets were collected. The objective of this project has been met as the data digester was found to have a large amount of data successfully. The collected tweets were also successfully formatted by the data digester and inserted into the database. The testing of the analytical engine shows that the Stanford POS tagger was able to identify the name of politician, political party or a state in India with a recall value of 1 and a precision value of 0.47. In order to crawl more relevant tweets relating to Indian politics, improvement should be made to how the twitter crawler crawls for tweets. Instead of keywords, the crawler should start crawling tweets from a list of twitter users which are very active in Indian politics. Currently the visualizer, operational database and twitter crawler reside on the same server. To improve performance of the overall system, two servers should be used; one for hosting the operational database and one for data warehousing.Bachelor of Engineering (Computer Engineering