An Experimental Evaluation of Continuous Testing during Development

Continuous testing uses excess cycles on a developer’s workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It is intended to reduce the time and energy required to keep code well-tested and prevent regression errors from persisting uncaught for long periods of time. This paper reports on a controlled human experiment to evaluate whether students using continuous testing are more successful in completing programming assignments. We also summarize users ’ subjective impressions and discuss why the results may generalize. The experiment indicates that the tool has a statistically signif-icant effect on success in completing a programming task, but no such effect on time worked. Participants using continuous testing were three times more likely to complete the task before the dead-line than those without. Participants using continuous compilation were twice as likely to complete the task, providing empirical sup-port to a common feature in modern development environments. Most participants found continuous testing to be useful and be-lieved that it helped them write better code faster, and 90 % would recommend the tool to others. The participants did not find the tool distracting, and intuitively developed ways of incorporating the feedback into their workflow