Configuration-aware regression testing: an empirical study of sampling and prioritization

Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Re-searchers have been creating techniques for testing config-urable software, but to date, only a little research has ad-dressed the problems of regression testing configurable sys-tems as they evolve. Whereas problems such as selective retesting and test prioritization at the test case level have been extensively researched, these problems have rarely been considered at the configuration level. In this paper we ad-dress the problem of providing configuration-aware regres-sion testing for evolving software systems. We use combina-torial interaction testing techniques to model and generate configuration samples for use in regression testing. We con-duct an empirical study on a non-trivial evolving software system to measure the impact of changes in configurations on testing effectiveness, and to compare the effectiveness of different configuration prioritization techniques on early fault detection. Our results show that configurations can have a large impact on fault detection and that prioritiza-tion of configurations can be effective. 1