Irreproducible benchmarks might be sometimes useful

Historically, benchmarks have been used for commercial purposes. A customer develops or selects a benchmark that generates a load that is considered to be typical for her/his applications. The benchmark is executed on various machines to find the most appropriate vendor and machine. In OS research, we use benchmarks for completely different purposes that can be subsumed under the headline “understanding a complex system”: 1. Rating: We run a benchmark, then we modify something in the examined system (hardware or software), and run the benchmark a second time. We use the observed performance difference between both benchmarks as a measure characterizing the effects of the modification. 2. Discovering experiments: We run a benchmark and trace, count, or measure certain events, e.g. cache misses, system calls, hits in the file cache, etc. With such experiments, we try to “understand” a system and to find a more or less sophisticated model for it. In the most ambitious case, such a model would allow us to predict the effects of modifying the system; in the simplest case, it should at least help us to identify bottlenecks