Scientific Benchmarking of Parallel Computing Systems Twelve ways to tell the masses when reporting performance results

Measuring and reporting performance of parallel computers con-stitutes the basis for scientific advancement of high-performance computing (HPC). Most scientific reports show performance im-provements of new techniques and are thus obliged to ensure repro-ducibility or at least interpretability. Our investigation of a strati-fied sample of 120 papers across three top conferences in the field shows that the state of the practice is lacking. For example, it is of-ten unclear if reported improvements are deterministic or observed by chance. In addition to distilling best practices from existing work, we propose statistically sound analysis and reporting tech-niques and simple guidelines for experimental design in parallel computing and codify them in a portable benchmarking library. We aim to improve the standards of reporting research results and initi-ate a discussion in the HPC field. A wide adoption of our minimal set of rules will lead to better interpretability of performance results and improve the scientific culture in HPC