Tracking Epidemics with Natural Language Processing and Crowdsourcing

The first indication of a new outbreak is often in un-structured data (natural language) and reported openly in traditional or social media as a new ‘flu-like ’ or ‘malaria-like ’ illness weeks or months before the new pathogen is eventually isolated. We present a system for tracking these early signals globally, using natural lan-guage processing and crowdsourcing. By comparison, search-log-based approaches, while innovative and in-expensive, are often a trailing signal that follow open reports in plain language. Concentrating on discovering outbreak-related reports in big open data, we show how crowdsourced workers can create near-real-time train-ing data for adaptive active-learning models, addressing the lack of broad coverage training data for tracking epi-demics. This is well-suited to an outbreak information-flow context, where sudden bursts of information about new diseases/locations need to be manually processed quickly at short notice