Accounting for Context and Lifetime Factors: A New Approach for Evaluating Regression Testing Techniques

Regression testing is an expensive testing process performed on modified software to provide confidence that the software behaves correctly, and that modifications have not impaired its quality. Regression testing is widely used in industry; however, it is often performed inadequately. As a result, software quality and reliability can decrease over the software\u27s lifetime. To address this problem, researchers have spent a great deal of effort creating and studying various methodologies for improving the cost-effectiveness of regression testing. To compare and assess such methodologies, researchers relied initially on analytical approaches. More recently, however, focus has shifted to empirical studies. Empirical studies of regression testing techniques have greatly expanded our understanding of techniques and the factors that affect them, but to date, these studies have also suffered from several limitations which limit the extent to which their results may generalize to practice. (1) Most studies have considered only a few context factors (characteristics of the environment or engineering processes that may affect technique performance). (2) Prior studies have calculated costs and benefits using a snapshot view in which results are considered strictly per system version; this approach, however, ignores the fact that methodologies may exhibit different cost-benefit tradeoffs when assessed across entire system lifetimes than when assessed relative to individual versions. (3) Previous studies have largely ignored cost-benefit tradeoffs, relying on comparisons strictly in terms of simple benefit and cost factors, using cost-benefit models that are naive in their handling of important revenue and cost components, or using metrics that render comparisons across specific types of techniques impossible. Limitations such as these make it difficult or impossible to accurately compare and assess regression testing methodologies relative to practical software engineering contexts. Moreover, they can lead researchers and practitioners to inaccurate conclusions about the relative cost-effectiveness of techniques in practice, or the suitability of particular techniques to particular engineering processes. This dissertation addresses these limitations. First, we surveyed the state of the art of empirical studies of regression testing techniques and identified problems with evaluation methods and processes, and problems related to infrastructure required for empirical studies. Second, we developed infrastructure to support empirical studies of regression testing considering a wide variety of software artifacts. Third, using the infrastructure developed in the second step, we conducted several initial empirical studies on regression testing techniques. Fourth, we developed a cost-benefit model to assess the cost-effectiveness of regression testing techniques considering system lifetime and context factors. Finally we conducted an empirical study, assessing regression testing techniques using these cost-benefit models. Through our work, we provide several important advantages for practitioners and researchers. For practitioners, we provide new practical understanding of regression test techniques. For researchers, we provide a new cost-benefit model that can be used to compare and empirically evaluate regression testing techniques, and that accounts for testing context and system lifetime factors. We identify problems involving infrastructure, and provide infrastructure that can help researchers conduct various controlled experiments considering a wide variety of software artifacts. Finally, we provide better understanding of empirical methodologies that can be used by other researchers to make further progress in this area