Timestamped state sharing for streaming data analytics

M.Phil.Streaming data analytics has gained a lot of attention in recent years. State access in most existing distributed stream processing systems is limited within each operator instance locally. However, in many advanced stream analytics workloads such as dynamic graph analytics and online learning, state sharing across operator instances, operators and dataflows will make application development much easier and stream processing more efficient. In addition, the streaming data records in these workloads are often timestamped, which further requires proper time semantics to be defined for both state update and state access.In this thesis, we identify the key challenges related to state management for processing these workloads, and propose a new state abstraction to address the limitations of existing systems. We then present the architecture and detailed design of Nova, a distributed stream processing system with native management of system-wide states. Nova supports fast concurrent state access, provides flexible consistency control and guarantees exactly-once processing, which allows users to build performant and robust complex streaming applications with minimal effort. Finally, We conduct extensive experiments of dynamic graph analytics, online learning and common streaming benchmarks to show that Nova's expressiveness relieves users' burden, and outperforms or achieves comparable performance of composite solutions or specialized systems.流式數據處理在最近幾年越來越受到關注。狀態的讀寫在大多數現有的分佈式流式處理系統中僅僅局限于一個物理算子的內部。但是，在很多複雜的流式數據處理的應用，如動態圖的分析以及在線機器學習等，在不同的物理算子、不同的邏輯算子以及不同的流之間共享狀態會使得應用的構建更加方便而且高效。並且，由於在這些應用中的流式數據往往帶有時間的信息，這就需要我們為狀態的更新和讀取定義合理的時間語義。在這篇論文中，我們指出在分佈式流式系統中處理這些應用時，對於狀態的管理具體有哪一些關鍵的挑戰，並且提出了一種新的狀態的抽象來解決現有流式系統地局限。我們構建了一個新的分佈式流式處理系統叫做Nova，它在系統內部原生地實現了系統範圍內可見的帶有時間信息的狀態管理。我們通過不同的實驗來驗證Nova的豐富的表達能力以及高效的性能。Liu, Zhi.Thesis M.Phil. Chinese University of Hong Kong 2019.Includes bibliographical references (leaves 84-94).Abstracts also in Chinese.Title from PDF title page (viewed on 10, May, 2021)