Probabilistic assessment of effectiveness of software testing for safety-critical systems

Abstract: This paper provides the results of experimental evaluation of effectiveness of pair-wise testing. We consider a non-traditional area of pair-wise testing application, namely using pair-wise approach for testing logical expressions. A tool has been developed for experimental numerical investigation. This tool generates faults of various specific types and applies the previously derived test sets to determine if the testing method detected those faults. The user can see how effective a particular test case was at discovering a particular fault type. We investigate fault detection for five various types of faults in 20 different logical expressions which contain from 5 to 14 logical variables each. Three different pair-wise test sets, generated by two different tools, are applied for each logical expression and each generated fault. The program calculates probabilities of faults revealed over single expressions, the entire test set, and between fault types. We compare our results with results by other researchers and provide recommendation for practical testing. The tool has been used for analysis of pair-wise testing, however, it can be also used to analyze other testing criteria