Configuration aware prioritization for regression testing

Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Configurability requires extra effort for testing because there is evidence that running the same test case on different configurations may detect different faults. Differentiating test cases and configurations as two independent factors for testing, we must consider not just which test case to utilize, but also which configurations. Ideally, an exhaustive testing approach would combine every test case with every possible configuration. But since the full configuration space of most software systems is huge, it is infeasible to test all possible configurations with all test cases. Instead, sampling techniques are applied. Despite successful sampling techniques, sometimes it is still costly to run only a configuration sample. In particular, the cost is magnified when new features and functionality are added as a system evolves, and the new version is regression tested. Regression testing is an important but expensive way to build confidence that software changes introduce no new faults as software evolves, and many efforts have been made to improve its performance given limited resources. For example, regression test selection and test case prioritization have been extensively researched, but they have rarely been considered for configurations. In this dissertation, we provide cost-effective prioritization techniques for regression testing configurable systems—configuration aware regression testing. Specifically, we first generalize the problem of configuration sampling, and systematically compare different sampling techniques for configurations in non-trivial software systems, across multiple consecutive versions. We then investigate different prioritization techniques for ordering configurations sampled. We also investigate the relative cost-benefits of prioritizing test cases and configurations, as two independent factors, by presenting a comprehensive method for prioritizing both test cases and configurations. These techniques and methods are evaluated through empirical studies. Finally, we extend or modify basic prioritization techniques for different practical environments that involve additional factors (e.g., cost) or constraints (e.g., without testing records of prior versions)