An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment

Continuous\ua0integration (CI) emphasizes quick feedback to developers. This is at odds with current practice of\ua0performance\ua0testing, which predominantely focuses on long-running tests against entire systems in production-like environments. Alternatively,\ua0software\ua0microbenchmarking attempts to establish a\ua0performance\ua0baseline for small code fragments in short time. This paper investigates the quality of\ua0microbenchmark\ua0suites\ua0with a focus on suitability to deliver quick\ua0performance\ua0feedback and CI integration. We study ten open-source libraries written in Java and Go with benchmark\ua0suite\ua0sizes ranging from 16 to 983 tests, and runtimes between 11 minutes and 8.75 hours. We show that our study subjects include benchmarks with result variability of 50% or higher, indicating that not all benchmarks are useful for reliable discovery of slowdowns. We further artificially inject actual slowdowns into public API methods of the study subjects and test whether test\ua0suites\ua0are able to discover them. We introduce a performance-test quality metric called the API benchmarking score (ABS). ABS represents a benchmark\ua0suite\u27s ability to find slowdowns among a set of defined core API methods. Resulting benchmarking scores (i.e., fraction of discovered slowdowns) vary between 10% and 100% for the study subjects. This paper\u27s methodology and results can be used to (1) assess the quality of existing\ua0microbenchmark\ua0suites, (2) select a set of tests to be run as part of CI, and (3) suggest or generate benchmarks for currently untested parts of an API