Experiences in Benchmarking of Autonomic Systems

Abstract. Autonomic computing promises improvements of systems quality of service in terms of availability, reliability, performance, se-curity, etc. However, little research and experimental results have so far demonstrated this assertion, nor provided proof of the return on invest-ment stemming from the efforts that introducing autonomic features re-quires. Existing works in the area of benchmarking of autonomic systems can be characterized by their qualitative and fragmented approaches. Still a crucial need is to provide generic (i.e. independent from busi-ness, technology, architecture and implementation choices) autonomic computing benchmarking tools for evaluating and/or comparing auto-nomic systems from a technical and, ultimately, an economical point of view. This article introduces a methodology and a process for defining and evaluating factors, criteria and metrics in order to qualitatively and quantitatively assess autonomic features in computing systems. It also discusses associated experimental results on three different autonomic systems