A framework for performance-oriented real-time stream processing over the cloud

© 2019 Tri Minh TruongStream processing is an in-memory computing paradigm that supports querying over continuous data streams with a process-once-arrival strategy. Compared to a store-then- process model, stream-based queries can be performed on a continuous basis over data arriving in real-time data streams. The key strength of stream processing is that it can provide faster insights and does not suffer from data storage and retrieval challenges, that can be quite wasteful when not all of the data being stored is ultimately useful. Given the potential for high velocity of data arrival, ensuring performance in stream processing is essential for a wide range of time-critical applications such as fraud detection and health surveillance. Data streams can be bursty in nature hence stream computing platforms need to be able to adapt to given data flow circumstances and the underpinning infrastructure support. Cloud computing provides users with scalable and flexible computing and storage resources in a pay-as-you-go manner. As such, Cloud computing offers many advantages for stream-based applications with time-varying workloads. Challenges exist however where those applications are expected to be offered at affordable cost and with a required Quality of Service (QoS) in a dynamic Cloud environment. The focus of this thesis is to realise a Cloud-based framework which covers the whole stream processing lifecycle: from deployment, monitoring and adapting system resources based on the specific performance demands of the application. Specifically, we investigate novel approaches for performance analysis and modelling to tackle the challenges related to resource provisioning, operator parallelism and scheduling associated with Cloud-based deployments of stream processing systems. In particular, this thesis makes the following contributions: -A strategy to estimate the resources needed when deploying stream processing systems and subsequently predicting the stability of such systems; -A strategy to analyse the expected performance of stream processing systems. An algorithm is designed and tested based on throttling the behaviour of streams to estimate the expected (effective) utilization levels achievable under various deployment strategies and the subsequent impact on the performance of the systems. -A performance-oriented scaling algorithm for stable, large-scale stream processing. The approach takes into account the behaviour of stream buffer queue lengths, as well as the effective utilization of system resources required to achieve effective throughput whilst minimizing latency and resources. -A performance and cost trade-off algorithm for stable, large-scale stream processing on the Cloud. This includes a strategy for deploying new plans that consider the variability of input and output rates of individual stream processing operators at run-time whilst factoring in the associated server start-up costs