Clash of the Titans: Tools and Techniques for Hunting Bugs in Concurrent Programs

In this work we focus on creating a benchmark suite of concurrent programs for various programming languages to evaluate the bug detection capabilities of various tools and techniques. We have compiled a set of Java benchmarks from various sources and our own efforts. For many of the Java examples we have created equivalent C # programs. All the benchmarks are available for download. In our multi-language benchmark suite we compare results from various tools: CalFuzzer, ConTest, CHESS, and Java Pathfinder. In Java Pathfinder we provide extensive results for state-less random walk, randomized depth-first search, and guided search using abstraction refinement. Using data from our study we argue that iterative context-bounding and dynamic partial order reduction are not sufficient to render model checking for testing concurrent programs tractable and sec-ondary techniques such as guidance strategies are required. As part of this work, we have also created a wiki to pub-lish benchmark details and data from various tools on those benchmarks to a broader research community