Automated Quality-Assurance Techniques for the Modernization of Software-Product Lines

Nowadays, quality assurance for highly configurable software has become increasingly important. One of the most established ways to ensure software quality is testing. For example, old legacy systems are enhanced (called retrofitting) to cope with new Industrie 4.0 demands and the variability of the environments. During retrofitting, changes to the systems must not introduce new bugs, and backward compatibility must be ensured. Additionally, software is often implemented as a software-product line (i.e., a family of similar but distinguishable products) to include the variability. This introduces three challenges for quality assurance in Industrie 4.0. First, testing single and often safety-critical products. Second, ensuring backward compatibility and quality assurance during retrofitting and lastly, to cope with configurability during testing. To automate the generation of test cases for quality assurance, different tools exist. Those tools usually extract test goals for a given program based on a coverage criterion (e.g., statement coverage) and try to create test cases to cover all test goals. Additionally, in different environments, the focus on efficiency (e.g., CPU time) or effectiveness (the number of bugs found) differs. For safety-critical systems (e.g., autonomous systems in Industrie 4.0), software quality is crucial. Therefore, the focus is on effectiveness. In other cases (e.g., certification of systems) the focus is on efficiency due to fixed requirements for testing. Although effectiveness and efficiency often obstruct each other, both, and especially the trade-off between both can be improved. However, currently few studies are concerned with the trade-off during testing and the impact of different test-case generation strategies on the trade-off.This thesis presents novel techniques to increase efficiency, effectiveness and optimize the trade-off for test-case generation. First, we focus on increasing efficiency for single products (e.g., for certification purposes). Since generating test cases for a single product often leads to many test goals, we developed new techniques for test-case generation with many test goals. First, we group test goals into partitions and execute a single reachability analysis for each partition. Second, we combine different analysis techniques to further increase efficiency.We evaluate different partitioning strategies and combine two different analysis techniques with different time limits to study the impact on efficiency. Next, we present novel techniques to increase effectiveness during regression testing (i.e., testing changes, e.g., during retrofitting). The goal of regression testing is to detect bugs introduced due to the changes. However, as bugs are not known beforehand, no test goal exists that guarantees to detect the bug. Still, some test goals lead to test cases with high chances to detect a bug. For example, test goals resulting in test cases leading to different output behaviors compared to the previous program version are usually more effective compared to test cases only traversing the program changes. The methodology developed in this thesis enables us to configure novel test-case generation strategies (and test-suite reduction) to tune the resulting test suite in terms of effectiveness, efficiency, and especially in terms of the trade-off. In our evaluation, we also measured the impact of different parameters of our methodology on effectiveness and efficiency. Lastly, to cope with the configurability of software-product lines, specific testing techniques are needed. One established technique to cope with the configurability during testing is sampling. Sampling derives a set of products from the product line and uses conventional testing techniques for each product. However, the effectiveness of the sample often remains unknown. To this end, we developed a technique based on mutation testing to measure the quality of sampling strategies based on effectiveness and efficiency. All techniques and methodologies developed during this thesis are motivated and explained on a running example and evaluated on real-world and synthetic programs