Validating search protocols for mining of health and disease events on Twitter

In the year of 2016, there were more than 24 million Indonesian twitter users sharing news, events, as well as personal feelings and experiences on Twitter. This study seeks to validate a search protocol of health-related terms using real-time Twitter data which can later be used to understand if, and how, twitter can reveal information on the current health situation in Indonesia. In this validation study of mining protocols, we extracted geo-located conversations related to health and disease postings on Twitter using a set of pre-defined keywords, assessed the prevalence, frequency and timing of such content in these conversations, and validated how this search protocol was able to detect relevant disease tweets. Groups of words and phrases relevant to disease symptoms and health outcomes were used in a protocol developed in the Indonesian language in order to extract relevant content from geo-tagged Twitter feeds. A supervised learning algorithm using Classification and Regression Trees was used to validate search protocols of disease and health hits comparing to those identified by a team of human experts. The experts categorized tweets as positive or negative in respect to health events. The model fit was evaluated based on prediction performance. We observed 390 tweets from historical Twitter feeds and 1,145,649 tweets from Twitter stream feeds during the period July 26th to August 1st, 2016. Only twitter hits with health related keywords in the Indonesian language were obtained. The accuracy of predictions of mined hits versus expert validated hits using the CART algorithm showed good validity with AUC beyond 0.8. Our study shows that monitoring of public sentiment on Twitter, combined with contextual knowledge about the disease, can detect health and disease tweets and potentially be used as a valuable real-time proxy for health events over space and time