Configurations everywhere: Implications for testing and debugging in practice

us.abb.com Many industrial systems are highly-configurable, complicat-ing the testing and debugging process. While researchers have developed techniques to statically extract, quantify and manipulate the valid system configurations, we conjecture that many of these techniques will fail in practice. In this paper we analyze a highly-configurable industrial applica-tion and two open source applications in order to quantify the true challenges that configurability creates for software testing and debugging. We find that (1) all three appli-cations consist of multiple programming languages, hence static analyses need to cross programming language barriers to work, (2) there are many access points and methods to modify configurations, implying that practitioners need con-figuration traceability and should gather and merge meta-data from more than one source and (3) the configuration state of an application on failure cannot be reliably deter-mined by reading persistent data; a runtime memory dump or other heuristics must be used for accurate debugging. We conclude with a roadmap and lessons learned to help prac-titioners better handle configurability now, and that may lead to new configuration-aware testing and debugging tech-niques in the future