Producing wrong data without doing anything obviously wrong

This paper presents a surprising result: changing a seemingly innocuous aspect of an experimental setup can cause a sys-tems researcher to draw wrong conclusions from an experi-ment. What appears to be an innocuous aspect in the exper-imental setup may in fact introduce a significant bias in an evaluation. This phenomenon is called measurement bias in the natural and social sciences. Our results demonstrate that measurement bias is signif-icant and commonplace in computer system evaluation. By significant we mean that measurement bias can lead to a per-formance analysis that either over-states an effect or even yields an incorrect conclusion. By commonplace we mean that measurement bias occurs in all architectures that we tried (Pentium 4, Core 2, and m5 O3CPU), both compilers that we tried (gcc and Intel’s C compiler), and most of the SPEC CPU2006 C programs. Thus, we cannot ignore mea-surement bias. Nevertheless, in a literature survey of 133 re-cent papers from ASPLOS, PACT, PLDI, and CGO, we de-termined that none of the papers with experimental results adequately consider measurement bias. Inspired by similar problems and their solutions in other sciences, we describe and demonstrate two methods, one for detecting (causal analysis) and one for avoiding (setup randomization) measurement bias