Meaningful and Practical Measures for Regression Test Reliability

I assure the single handed composition of this master’s thesis only supported by declared resources. Munich, September 13, 2013 Rainer Niedermayr Successful software systems are subject to continuous maintenance and further development. Au-tomated tests can help to detect faults caused by changes at an early stage and therefore play an important role in the software development process. In the practice, the code coverage metric is often used as a criterion to evaluate the reliability of test suites with focus on regression errors. However, the code coverage expresses only, which portion of a program has been executed by tests, but not, how reliable the tests actually are. This thesis comprises on the one hand an analysis, how meaningful the code coverage is, and on the other hand the development of further criteria and measures to reveal inadequately tested code chunks. The measures were implemented in a prototype for Java applications based on the concept of mutation testing. It was a requirement that the execution of the analysis is applicable even to larger systems in reasonable time. Within a case study, 15 open source projects of differen