Findings from the First Annual File and Storage Systems Benchmarking Workshop

A growing consensus in the community of file and storage system researchers and practitioners believe that the quality of benchmarking must be improved significantly. We have found that there is often too little scientific methodology or statistical rigor behind current benchmarking, which is largely done ad-hoc. In response, with the goal of improving the quality of performance evaluation in the field, we held the Storage and File Systems Benchmarking Workshop on May 19, 2008 at the University of California Santa Cruz. It was sponsored by the Storage Systems Research Center (SSRC, www.ssrc.ucsc.edu). This workshop brought together top researchers and practitioners from industry and academia, repre-senting all levels of the storage stack, along with statisticians and other interested parties. The main goals of the workshop were to educate everyone on the problems at hand, and discuss possible solutions. Participants presented relevant topics, and there was much interaction and discussion. The goal of this effort is improving the scientific and statistical methodologies used. This goal requires little research in the field, but does require educating both those who conduct performance evaluations and those who analyze results. It also requires program committees and reviewers to raise the bar on the quality of performance evaluations in accepted papers. A longer-term goal is to have computer scientists embrace the rigor of the other sciences. It is essential to be able to validate the results of others. Without it, it is meaningless to compare the performance of two systems. All presentations and slides are available at www.ssrc.ucsc.edu/wikis/ssrc/BenchmarkingWorkshop08/. Why File and Storage System Benchmarking is Difficult Erez Zadok, the workshop’s chair, began with an overview of the storage stack, some complexities that make benchmarking these systems a difficult task, and some examples of poor benchmarking practices. Some of the factors contributing to the complexity are: • Storage variety: storage does not consist only of a single local hard drive. Other types includ