Do Mutation Reduction Strategies Matter?

Abstract—Mutation analysis is a well-known, but computa-tionally intensive, method for measuring test suite quality. While multiple strategies have been proposed to reduce the number of mutants, there is inconclusive evidence for their utility due to the limited number and size of programs used for validation, and a lack of comprehensive comparative studies. Traditional evaluation criteria for mutation reduction also rely on mutation-adequate suites, which are rare in practice. We propose novel criteria for evaluating reduction strategies for non-mutation-adequate test suites, directly linked to the actual use of mutation analysis during development — to ensure that tests check for many different possible faults. We evaluate using both these criteria and the traditional criteria with 201 real-world projects, and show that the popular strategies — operator selection, and stratified sampling (on operators or program elements) — are at best marginally better than random sampling, and are often worse. I