Mutation testing: Clustering the mutants

Software testing is a crucial part of the software engineering process. A part of software testing is building test suites which contain unit tests. These unit tests are written by developers. As projects grow the test suite grows along. Maintaining and monitoring these test suites is important as they influence the cost of maintenance. For example, a project with a smaller test suite may have a higher maintenance cost as the effect of change has to be measured manually, which can be a time consuming process. Monitoring the quality of test code has shown that tests with lower quality lead to more defect-prone production code. A technique to measure the quality of test code is by measuring the test effectiveness. Measuring the effectiveness of test suites detecting faults is part of the monitoring process. A technique to measure the test effectiveness is mutation testing. Mutation testing is a computationally expensive technique that requires the ability to run tests. Different techniques have been proposed to reduce the cost of mutation testing. One of these techniques is the clustering of mutants. By clustering mutants we can execute less mutants to reduce the cost. Our research consists of two parts in which we cluster mutants. The first part consists of a white box approach and the second part of a black box approach. We conducted a state of the art comparison between three mutation testing tools for Java. Based on this research we selected a mutation testing tool to use in our research. We identified characteristics to represent the mutants such that we can cluster them. For our white box approach we used hierarchical clustering and for our black box approach we trained the fuzzy c-means model. We calculated a weighted mutation score and compared this with the mutation score of a full set of mutants executed. Results show that with hierarchical clustering we can reduce the amount of mutants executed while maintaining the effectiveness. The clusters generated by the machine learning model were less accurate and showed a significant decrease in effectiveness. The machine learning model did not perform as well as the hierarchical clustering. However, more can be done by researching different approaches for training the FCM model and finding the right parameters