Experimental program analysis

Program analysis techniques are used by software engineers to deduce and infer characteristics of programs for software-engineering-related tasks. Recently, some program analysis techniques have been designed to leverage characteristics of traditional experimentation. An examination of these techniques suggests that a new form of program analysis technique can be created by incorporating characteristics of experimentation. To date, however, there has been little formal recognition by the software engineering community of this form of program analysis, or the implications of using it. This dissertation presents experimental program analysis as a unique paradigm for conducting program analysis. We define this paradigm by building on principles and methodologies underlying the use of experimentation in other fields. This work offers four primary contributions. First, we provide definitions of experimental program analysis, illustrate them by example, and describe several intriguing differences between experimental program analysis and the use of experimentation in other research fields. Second, we survey the research literature for experimental program analysis techniques to offer insights into their existence and diversity—both in terms of the techniques themselves and the problem domains in which they operate. Third, we explore the applicability of experimental program analysis in three software engineering problem domains to provide a formative assessment of the capabilities of these techniques, and how they compare—in terms of cost-effectiveness, utility, and capabilities—to non-experimental baseline techniques. Fourth, we explore the use of experimental program analysis in a large software development setting, and present a case study to investigate the use of experimental program analysis in that setting. The contributions from the foregoing work support the conjecture that the applicability of experimental program analysis could be substantial. In particular, we expect that this work will expose the research community to use of experimental program analysis techniques, identify domains in which experimental program analysis might be useful, suggest many opportunities for improvements to existing techniques, and promote the use of experimentation in program analysis in order to confront program analysis programs in new and innovative ways. Accordingly, we believe that the experimental program analysis paradigm offers a promising new direction for program analysis research