Automated Detection of Performance Regressions Using Statistical Process Control Techniques

The goal of performance regression testing is to check for performance regressions in a new version of a software sys-tem. Performance regression testing is an important phase in the software development process. Performance regres-sion testing is very time consuming yet there is usually lit-tle time assigned for it. A typical test run would output thousands of performance counters. Testers usually have to manually inspect these counters to identify performance re-gressions. In this paper, we propose an approach to analyze performance counters across test runs using a statistical pro-cess control technique called control charts. We evaluate our approach using historical data of a large software team as well as an open-source software project. The results show that our approach can accurately identify performance re-gressions in both software systems. Feedback from practi-tioners is very promising due to the simplicity and ease of explanation of the results. 1