Quantifying the effectiveness of testing via efficient residual path profiling

Writing correct programs is hard. Proving that they are correct is even harder. Consequently, testing is extensively used for un-covering bugs in large, complex software. Since testing software exhaustively is infeasible, well designed regression test suites aim to anticipate all reasonable software usage scenarios and generate test cases that exercise those behaviors. Unfortunately, testers to-day have no way of knowing how much of real-world software usage was untested by their regression suite. While collecting path profiles of deployed software would provide this information, profiling over-heads preclude this. This often results in released software shipping with bugs that could have been detected with a better test suite. Recent advances in low-overhead path profiling provide the oppor-tunity to rectify this deficiency and perform residual path profiling on deployed software. Residual path profiling identifies all paths executed by deployed software that were untested during software development. We extend prior research to perform low-overhead in-terprocedural path profiling. We demonstrate experimentally that low-overhead path profiling, both intraprocedural and interproce-dural, provides valuable quantitative information on testing effec-tiveness. We also show that residual edge profiling is inadequate as a significant number of untested paths include no new untested edges