Comparing module- and class-level regression test selection in continuous integration

Developers rely on regression testing in their continuous integration (CI) environment to find if their changes introduce faults. While regression testing is widely practiced, it is very costly. Regression test selection (RTS) reduces the cost of regression testing by running only the tests that are affected by the changes. Industry has adopted module-level RTS running in a CI environment. Researchers have recently proposed class-level RTS but did not evaluate it in a CI environment. In this work, we compare module-level and class-level RTS techniques in a CI environment. Specifically, we evaluate GIB, a module-level RTS technique, and Ekstazi, a class-level RTS technique, on Travis CI, a popular CI service for open-source projects. We also propose a hybrid RTS technique, GIBstazi, that combines aspects of the module-level and class-level RTS techniques. GIBstazi is safer than Ekstazi because GIBstazi tracks non-source-code changes while Ekstazi does not. We evaluate all the techniques on Travis CI using real, historical commits that were built on Travis CI. Our results show that all the RTS techniques do save testing time compared to running all tests (RetestAll) in a CI environment, but the savings in the end-to-end time for a full build is not as high as shown in previous work in a local environment running with a dedicated machine due to the extra overhead in a CI environment. Moreover, we are the first to inspect test failures from RetestAll builds. We find that these failures are often not selected by the RTS techniques, but almost all test failures missed by the RTS techniques are from so called flaky tests, which can pass or fail nondeterministically regardless of the code changes. The overall results show that RTS can be beneficial for the developers, hybrid GIBstazi gives the best trade-off, and RTS not only saves time but also avoids misleading developers by flaky test failures