An Empirical Study of Off-by-one Loop Mutation

Context: Developing test cases that are measurably effective in finding faults in programs is a very challenging research problem. Mutation testing, a prominent technique developed to address this challenge, often becomes com- putationally too expensive for practical use due to the very large number of mutants that need to be analyzed. Objective: This paper evaluates the impact of One-by-one (OBO) loop mutation in reducing the cost of mutation analysis and investigates this technique\u27s effectiveness in measuring the strength or weakness of test suites. Method: A set of Java and C programs have been used to generate both OBO and traditional mutants. Mutation scores are computed and analyzed for both sets of mutants. An analysis of first order vs. higher order loop mutations have also been performed. Results: On average, 89.15% fewer mutants are generated by OBO op- erator in comparison to traditional operators while the two sets of muta- tion scores still remain highly positively correlated (correlation coefficient of .9228) indicating the usefulness of OBO operator in measuring test suite\u27s ef- fectiveness of finding faults in programs. We also investigate the relationship between first order OBO mutation (FOM) and their corresponding higher order mutations (HOM). We have found that OBO HOMs do not subsume their corresponding FOMs. Conclusion: We conclude that One-by-one (OBO) loop mutant operator, which targets specific program elements for mutation, can greatly reduce the number of mutants generated, and thus make the mutation analysis relatively inexpensive and practical while still being capable of providing useful measurement of the strength or weakness of a test suite. Our investigation into the relationship between higher order OBO mutants (HOM) and first order OBO mutants (FOM) has revealed that OBO HOMs usually do not add any value to the mutation analysis over the corresponding FOMs