Experimental design for comparing static concurrency analysis techniques

ABSTRACT formally, experimentation can help develop estimates of Software engineering has suffered from a shortage of empirical studies. We recently undertook a study in an area that seems well suited to empirical investigation, comparing the performance of several static analysis tools for evaluating properties of concurrent software. During the course of that study, we encountered a number of significant issues that make designing a sound and unbiased study surprisingly difficult. It is these issues, and the tradeoffs they necessitate, that are the focus of this paper. average case performance for each of the techniques. In addition to performance, experimentation can provide information about the types of properties and programs each tool can verify, the accuracy of the analysis results, and the frequency of failure. Corbett [4, 5] carried out empirical studies comparing the performance of several tools in detecting deadlock in Ada tasking programs. Our goal was to extend Corbett's work by adding data flow analysis to the repertoire of techniques, by examining program-specific properties in addition to deadlock, by considering analysis accuracy and other measures of performance, and by carrying out careful statistical analysis of the results to check for various biases and to quantify the predictive power of our results. Thus, we undertook an experimental evaluation of several of the major analysis techniques as applied to many of the concurrency analysis programs that have appeared in the literature [2]