Fault-tolerant stream processing using a distributed, replicated file system

We present SGuard, a new fault-tolerance technique for dis-tributed stream processing engines (SPEs) running in clus-ters of commodity servers. SGuard is less disruptive to nor-mal stream processing and leaves more resources available for normal stream processing than previous proposals. Like several previous schemes, SGuard is based on rollback recov-ery [18]: it checkpoints the state of stream processing nodes periodically and restarts failed nodes from their most recent checkpoints. In contrast to previous proposals, however, SGuard performs checkpoints asynchronously: i.e., opera-tors continue processing streams during the checkpoint thus reducing the potential disruption due to the checkpointing activity. Additionally, SGuard saves the checkpointed state into a new type of distributed and replicated file system (DFS) such as GFS [22] or HDFS [9], leaving more memory resources available for normal stream processing. To man-age resource contention due to simultaneous checkpoints by different SPE nodes, SGuard adds a scheduler to the DFS. This scheduler coordinates large batches of write requests in a manner that reduces individual checkpoint times while maintaining good overall resource utilization. We demon-strate the effectiveness of the approach through measure-ments of a prototype implementation in the Borealis [2] open-source SPE using HDFS [9] as the DFS. 1