Managing and understanding distributed stream processing

Present-day computing systems have to deal with a continuous growth of data rate and volume. Processing these workloads should introduce as little latency as possible. Today's stream processors promise to handle large volumes of data while providing low-latency query results. In practice however, their computational model, variations of the workload, and the lack of tools for programmers can lead to situations where the latency increases significantly. The reason for this lies in the design of today's stream processing systems. Specifically, stream processors do not supply meaningful information for debugging root causes of latency problems. Additionally, they have inadequate controllers to automate resource management based on workload properties and requirements of the computation. Lastly, their reconfiguration mechanisms are not compatible with low-latency query processing and cause a significant latency increase while reconfiguring. Solving these problems is crucial to enable users, programmers and operators to maximize the effectiveness of stream processing. To solve these problems, we present three complementing solutions. We first propose a method to identify factors influencing query latency, using detailed internal measurements combined with knowledge of the computational model of the stream processor. Then, we use system-intrinsic measurements to design and implement an automatic scaling controller for scale-out distributed stream processors. It makes fast and accurate scaling decisions with minimum delay. Lastly, we propose a scaling mechanism that reduces downtime during reconfigurations by orders of magnitude. The mechanism achieves this by interleaving fine-grained configuration updates and data processing. The solutions presented in this thesis help designers of stream processors to better optimize for low-latency processing, and users to increase their query performance by providing better metrics and automating operational aspects. We think this is an important step towards efficient stream processing