A big data infrastructure for real-time traffic analytics on the cloud

© 2019 Yikai GongWith the increasing urbanisation occurring globally, cities are facing unprecedented challenges. One major challenge is related to traffic and the increasingly common congestion issues that arise in cities. At the same time, digital data is being created across all walks of life by industry, governments and society more generally. The term "big data'' has now entered common vernacular. Big data can include officially captured data, e.g. from traffic measurement systems from government organisations such as VicRoads in Australia, as well as other forms of data generated by the population at large, e.g. social media. This thesis explores the unique characteristics of traffic related data and focuses on the development and evaluation of an underpinning Cloud-based platform that can tackle some of the unique big data challenges related to such data. In particular, the thesis focuses on challenges related to the volume, velocity and variety of traffic data. We explore how different forms of data including official sensor data such as the Sydney Coordinated Adaptive Traffic System (SCATS) that is widely rolled out across Victoria and supported by VicRoads can be processed in real time, as well as how social media data such as Twitter can be used as a cheaper proxy for SCATS to better understand traffic in cities. We also develop novel real-time clustering algorithms that tackle the unique spatial and temporal aspects of traffic related data