An Experimental Evaluation of PIT’s Mutation Operators

Mutation testing is a fault-finding software testing technique that creates mutants by injecting a syntactic change into the source code. This technique operates by evaluating the ability of a testsuite to detect mutants, which is used to asses the quality of a test suite. The information from mutation testing can be used to improve the quality of a test suite by developing additional test cases. It is a promising technique, but the expensive nature of mutation testing has prevented its wide spread, practical use. One approach to make mutation testing more efficient is selective mutation testing. This approach proposes to use a subset of the available mutation operators, thereby reducing the number of mutants generated which will lead to a reduced execution time. According to theory many mutants are redundant and do not provide any useful information in the development of additional test cases. This study will evaluate PIT, a mutation testing tool, by using selective mutation testing on five programs, and compare the effectiveness of PIT with MuJava. The results showed that PIT’s default operators generated a small and effective set of mutants and at the same time giving a satisfying mutation score. However, there was no significant difference in mutation score produced by the different sets of operators. The test suites adequate for PIT managed to detect 75-85% of MuJava’s mutants, which is relatively good when taken into consideration that PIT generated less than half as many mutants