Karibu: A Flexible, Highly-Available, and Scalable Architecture for Urban Data Collection

Collecting data in a reliable and scalable way from a broad variety of Internet of Things (IoT) sensors requires significant effort, leading many researchers to use ad hoc data collection and storage systems that are unable to handle the quantity of data generated in dense urban environments, or to support long-term studies. In this paper, we present Karibu: a data collection architecture and reference implementation designed to meet the needs of the urban IoT community. Karibu enables sensor data collection from diverse sources and reliably stores the data in a scalable backend subsystem, allowing users to more efficiently pursue their research goals. We lay out the guiding principles of the Karibu architecture, including the data integrity and the modifiability design principles. We then detail our open source reference implementation of the architecture, which includes flexible and configurable client-side modules supporting the collection of data from heterogeneous sources and data types. Finally, we present our experiences using Karibu in two urban research projects: trip and transportation mode detection on smartphones, and exploring resource consumption in a highly-instrumented college dormitory