Use of deterministic execution in testing and debugging of distributed software

Large-scale networks are among the most complex software infrastructures in existence. Unfortunately, the increasing complexity of its software requirements leads to a rich variety of nondeterministic failure modes and anomalies. Research on testing and debugging modern distributed software has focused on designing comprehensive record and replay systems, but the large volumes of recordings often hinder the efficiency and scalability of these designs. Here, we argue for a different approach. Namely, we take the position that deterministic network execution would vastly simplify the distributed software testing and debugging process. This thesis presents the design and implementation of a network architecture for interactive testing and debugging that provides deterministic network execution of distributed software in highly distributed and dynamic environments. We design efficient and scalable algorithms for both control and data traffics in modern wide-area networks. In addition, we further describe several interactive primitives to reduce the time and effort on testing and debugging of large-scale distributed software. We demonstrate our system's advantages by analyzing nondeterministic ordering and timing bugs in popular real-world distributed software, XORP, Quagga, and lighttpd. Using Rocketfuel topologies and traffic data from a Tier-1 backbone, we show that our design is practical and scalable for interactive fault diagnosis in large networks