PerfDB + PerlML: Enabling Big Data-Driven Research On Fine-Grained Performance Phenomena

The long-tail latency problem is a well-known problem in large-scale system topologies like cloud platforms. Long-tail latency can lead to less predictable system performance, degraded quality of experience and potential economic loss. Previous research has focused on coarse- grained, symptomatic treatments like redundant request executions to mitigate tail latency and its effects. Instead, we propose studying these performance bugs systematically and addressing their underlying root cause. The millibottleneck theory of performance bugs provides a testable hypothesis for explaining at least some requests comprising the latency long tail. The theory posits that transient performance anomalies cause a non-negligible number of requests to complete in seconds, called Very Long Response Time Requests (VLRT), instead of tens of milliseconds like most other requests. In this dissertation, we enable the systematic evaluation of the millibottleneck theory across a big data-scale experimental data collection. First, we present perftables, a performance log parser, that extracts resource monitoring data across a wide variety of hardware and software configurations. Secondly, we use our data management system, PerfDB, to load and integrate fine-grained system performance data from approximately 400 experiments. We conduct the first-generation population study of VLRT, and our data support millibottlenecks inducing VLRT through CTQO (Cross-Tier Queue Overflow). We also enable the study of a second latency class called Less Long Requests (LLRs). Finally, we present our ensemble-based, supervised machine learning system, PerfML, that handles data characterized by heterogenous feature space and hierarchical, imbalanced classes—characteristics inherent to the data needed to study millibottlenecks and latency performance bugs. The analytics results from PerfML demonstrate its ability to isolate different kinds of millibottlenecks across a range of systems and configurations with high recall and acceptable precision.Ph.D