Fingerprinting the datacenter: automated classification of performance crises

Contemporary datacenters comprise hundreds or thousands of machines running applications requiring high availability and responsiveness. Although a performance crisis is easily detected by monitoring key end-to-end performance indica-tors (KPIs) such as response latency or request throughput, the variety of conditions that can lead to KPI degradation makes it difficult to select appropriate recovery actions. We propose and evaluate a methodology for automatic classification and identification of crises, and in particular for detecting whether a given crisis has been seen before, so that a known solution may be immediately applied. Our ap-proach is based on a new and efficient representation of the datacenter’s state called a fingerprint, constructed by statis-tical selection and summarization of the hundreds of perfor-mance metrics typically collected on such systems. Our eval-uation uses 4 months of trouble-ticket data from a produc-tion datacenter with hundreds of machines running a 24x7 enterprise-class user-facing application. In experiments in a realistic and rigorous operational setting, our approach pro-vides operators the information necessary to initiate recov-ery actions with 80 % correctness in an average of 10 min-utes, which is 50 minutes earlier than the deadline provided to us by the operators. To the best of our knowledge this is the first rigorous evaluation of any such approach on a large-scale production installation. 1