Configuration-aware regression testing: an empirical study of sampling and prioritization

Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Re-searchers have created techniques for testing configurable software, but to date, only a little research has addressed the problems of regression testing configurable systems as they evolve. Whereas problems such as selective retesting and test prioritization at the test case level have been exten-sively researched, these problems have rarely been consid-ered at the configuration level. In this paper we address the problem of providing configuration-aware regression testing for evolving software systems. We use combinatorial interac-tion testing techniques to model and generate configuration samples for use in regression testing. We conduct an empir-ical study on a non-trivial evolving software system to mea-sure the impact of configurations on testing effectiveness, and to compare the effectiveness of different configuration prioritization techniques on early fault detection during re-gression testing. Our results show that configurations can have a large impact on fault detection and that prioritization of configurations can be effective