Achieving Scalable and Reliable Non-Intrusive Failure Reproduction in Distributed Systems by Enhancing the Event Chaining Approach

Complex and unforeseen failures in distributed systems must be diagnosed and replicated so developers can understand the underlying problem and verify the resolution. Unfortunately, failure reproduction is unpredictable and time-consuming, often leading to costly service outages. Pensieve is a tool that automates failure reproduction by deploying a novel static analysis approach, Event Chaining (EC), which iteratively explains causal dependencies from the failure symptom while avoiding simulating the entire execution by skipping likely irrelevant instructions, which addresses the cause of poor scalability in existing approaches like symbolic execution. Despite its aggressive design, EC is plagued by combinatorial explosion. This thesis investigates ECâ s poor scalability and presents a redesign that enables EC to scale for complex failures. Further, this thesis presents a feedback mechanism that identifies instructions initially skipped by EC but are in fact relevant to the failure. Finally, this thesis presents a design that enables deterministically reproduction of concurrency failures.M.A.S