A study of the influence of assertions and mutants on test suite effectiveness

Test suite effectiveness is measured by assessing the portion of faults that can be detected by tests. To precisely measure a test suite’s effectiveness, one need to pay attention to both tests and the set of faults used. Code coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been shown to be highly correlated with effectiveness. However, previous studies treat test methods as the smallest unit of interest, and ignore potential factors influencing the correlation between test suite size and test suite effectiveness. We propose to go beyond test suite size, by investigating test assertions inside test methods. First, we empirically evaluate the relationship between a test suite’s effectiveness and the (1) number of assertions, (2) assertion coverage, and (3) different types of assertions. We compose 6,700 test suites in total, using 24,000 assertions of five real-world Java projects. We find that the number of assertions in a test suite strongly correlates with its effectiveness, and this factor positively influences the relationship between test suite size and effectiveness. Our results also indicate that assertion coverage is strongly correlated with effectiveness. Second, instead of only focusing on the testing side, we propose to investigate test suite effectiveness also by considering fault types (the ways faults are generated) and faults in different types of statements. Measuring a test suite’s effectiveness can be influenced by using faults with different characteristics. Assessing test suite effectiveness without paying attention to the distribution of faults is not precise. Our results indicate that fault type and statement type where the fault is located can significantly influence a test suite’s effectiveness.Applied Science, Faculty ofElectrical and Computer Engineering, Department ofGraduat