Benchmarks - are they really useful?

Benchmarking is an important commercial tool, which can be used for quantifying the performance of computer and network equipment. Furthermore, benchmarks are potentially valuable as part of curriculum studies in computer and network technology. Benchmarks may be of value to support the understanding of different architectural features and their effect on equipment performance. In effect the benchmarking results may provide a tangible metric that can be related directly not only to various architectural features but also the interactions between different levels in the memory hierarchy, hence providing a method of quantifying different performances of differing computer architectures and configurations. In this context a wide range of benchmarks were tested using the criteria of: repeatability, comparability, consistency, use of meaningful units etc. The criteria selected are based on the fundamental principles of measurement science. Our results clearly indicated that different benchmarking suites gave significantly different results for the same equipment. Also each benchmark failed to give consistent results, when compared to other benchmarks, even on identical equipment. Furthermore many of the benchmarks provided performance data in arbitrary units that were difficult to relate to expected changes in performance. In effect every benchmark tested failed to meet the evaluation criteria. The authors offer an alternative benchmarking method that is designed to meet such criteria and experimental work to date indicates some success using this new metric. This paper presents results of this work and gives recommendations regarding the use of benchmarks in computer education courses