Sequential Statistical Procedures for Approving Test Sets Using Mutation-Based Software Testing

Mutation analysis is a well-studied method of measuring test-case adequacy. Mutation analysis involves the mutation of a program by introduction of a small syntactic change in the software. Existing test data sets are then executed against all these mutant programs. If the test data set is adequate for testing the original program, it will distinguish all of the incorrect mutant programs from the original program. As an ad-hoc procedure, a stopping criterion is conventionally based on a given "Y% of the mutants to be distinguished" with a certain "confidence level of X%" over a multiplicity of random test cases. Alternatively, we propose a Bayes sequential procedure for testing H 0 : p = p 1 (acceptable fraction of live mutants to demonstrate good quality) vs. HA : p = p 2 (unacceptable fraction of live mutants to demonstrate bad quality). This derives a sequential probability ratio testing (SPRT) that is the most economical sampling scheme with given prior probabilities, decision and ..