Design, implementation, and evaluation of the linear road benchmark on the stream processing core

Stream processing applications have recently gained signifi-cant attention in the networking and database community. At the core of these applications is a stream processing en-gine that performs resource allocation and management to support continuous tracking of queries over collections of physically-distributed and rapidly-updating data streams. While numerous stream processing systems exist, there has been little work on understanding the performance charac-teristics of these applications in a distributed setup. In this paper, we examine the performance bottlenecks of streaming data applications, in particular the Linear Road stream data management benchmark, in achieving good performance in large-scale distributed environments, using the Stream Pro-cessing Core (SPC), a stream processing middleware we have developed. First, we present the design and implementation of the Linear Road benchmark on the SPC middleware. SPC has been designed to scale to tens of thousands of processing nodes, while supporting concurrent applications and mul-tiple simultaneous queries. Second, we identify the main performance bottlenecks in the Linear Road application in achieving scalability and low query response latency. Our results show that data locality, buffer capacity, physical al-location of processing elements to infrastructure nodes, and packaging for transporting streamed data are important fac-tors in achieving good application performance. Though we evaluate our system primarily for the Linear Road appli-cation, we believe it also provides useful insights into the overall system behavior for supporting other distributed and large-scale continuous streaming data applications. Finally, we examine how SPC can be used and tuned to enable a very efficient implementation of the Linear Road application in a distributed environment. ∗This work was done while the author was an intern at IBM T. J. Watson Research Center