Optimizing Continuous Integration by Dynamic Test Selection

Continuous integration (CI) is widely used in modern software engineering. However, it is an expensive practice. Some proposed approaches only focus on either intra- or inter-build cost reduction. In this paper, we propose an adaptive technique for dynamic test selection DTS, which combines intra- and inter-build cost reduction techniques. DTS uses build features to construct machine learning models to predict the probability of a specific build failure and transform the probability into the necessary test proportion, with respect to a selected test case prioritization technique. Based on the output of prediction model, it thus selects a prioritized test suite and a variable proportion of test cases with respect to a build. We constructed a large-scale dataset with approximately 115,000 builds, and conducted a controlled experiment using the dataset. The experiment shows that DTS outperforms existing techniques significantly. It detects 19.9% to 32.5% more failed test cases, compared with state-of-the-art techniques evaluated in the experiment. At the same time, DTS performs better than all three existing peer techniques on approximately 47% of projects. Moreover, the experiment also shows that our failure prediction model has an improvement of 0.15 in Area Under Curve (AUC), compared to prior machine learning models