Under review by NSDI 2006. Please do not cite or circulate without permission! Pip: Detecting the unexpected in distributed systems

Bugs in distributed systems are often hard to find. Many bugs reflect discrepancies between a system’s behavior and the programmer’s assumptions about that behavior. We present Pip, an infrastructure for comparing actual be-havior and expected behavior to expose structural errors and performance problems in distributed systems. Pip al-lows programmers to express, in a declarative language, expectations about the system’s communications struc-ture, timing, and resource consumption. Pip includes sys-tem instrumentation and annotation tools to log actual sys-tem behavior, and visualization and query tools for ex-ploring expected and unexpected behavior. Pip allows a developer to quickly understand and debug both familiar and unfamiliar systems. We applied Pip to several applications, including FAB, SplitStream, Bullet, and RanSub. We generated most of the instrumentation for all four applications automatically. We found the needed expectations easy to write, starting in each case with automatically generated expectations. Pip found unexpected behavior in each application, and helped to isolate the causes of poor performance and in-correct behavior.