Lessons Learned During the Development of the CapoOne Deterministic Multiprocessor Replay System ∗

Current schemes for deterministic replay of parallel applica-tions can be of great help for programmers. Software-only replay systems, while great at operating at the abstraction of user applications, incur large CPU overheads. Hardware-only systems produce minimal, if any, CPU overhead, but because their focus is the low-level hardware primitive that records and replays the memory access interleaving, they do not integrate well with user applications. The limitations of both the software- and hardware-only schemes render them impractical for most production uses. Capo is a hardware-assisted deterministic replay frame-work that combines the performance of hardware-only schemes with the flexibility of the software-only ones. CapoOne is the first implementation of the Capo framework. It has modest overheads and shows that practical determin-istic replay of production parallel applications is feasible. This paper brings together a set of lessons learned during the development of the CapoOne prototype so that designers of future hardware-assisted deterministic replay systems can apply them, lessening their pain. 1