Fast Failure Recovery in Distributed Graph Processing Systems

Distributed graph processing systems increasingly require many compute nodes to cope with the requirements imposed by contem-porary graph-based Big Data applications. However, increasing the number of compute nodes increases the chance of node fail-ures. Therefore, provisioning an efficient failure recovery strategy is critical for distributed graph processing systems. This paper pro-poses a novel recovery mechanism for distributed graph processing systems that parallelizes the recovery process. The key idea is to partition the part of the graph that is lost during a failure among a subset of the remaining nodes. To do so, we augment the exist-ing checkpoint-based and log-based recovery schemes with a par-titioning mechanism that is sensitive to the total computation and communication cost of the recovery process. Our implementation on top of the widely used Giraph system outperforms checkpoint-based recovery by up to 30x on a cluster of 40 compute nodes. 1