Investigating test selection techniques for scientific software using Hook’s mutation sensitivity testing

AbstractRecent research by Hook and Kelly into Mutation Sensitivity Testing provided a test analysis technique suitable for the challenges associated with testing scientific software. Specifically, Hook and Kelly’s work focused on overcoming two challenges associated with scientific software — the frequent lack of an oracle and dealing with the often present culmination of model refinements and calculations of the science associated with the software. Their research was specific to the detection of silent faults within scientific software made even more difficult to detect given the aforementioned two challenges. The Mutation Sensitivity Testing technique is built upon the mutation testing foundation laid by De Millo et al in their 1978 paper.To demonstrate how Mutation Sensitivity Testing could be used to improve both tests and testing practices Hook conducted a series of several small experiments in which eight functions were mutated and executed using 1155 test cases to perform a cursory evaluation into the effectiveness and efficiency of the test cases. As a follow-on to their work, Hook and Kelly’s initial results are analyzed in detail to extract any noticeable patterns in regards to the test efficiency of the tests. This initial research by Hook and Kelly suggested that only a few individual tests are needed to detect most of the code faults.The statistical analysis of their work suggests that for four of the eight functions, randomly selected inputs detected a higher number of mutants than test cases developed with “hand-picked” inputs. The statistical analysis supporting this claim are contained within this paper. Armed with this knowledge, the authors pressing forward with the analysis of random tests for an additional set of functions to investigate those testing techniques that would allow software testers to better select successful tests for scientific software