Diagnosing performance problems by visualizing and comparing system behaviours

Spectroscope is a new toolset aimed at assisting developers with the long-standing challenge of performance debugging in dis-tributed systems. To do so, it mines end-to-end traces of request processing within and across components. Using Spectroscope, developers can visualize and compare system behaviours between two periods or system versions, identifying and ranking various changes in the flow or timing of request processing. Examples of how Spectroscope has been used to diagnose real performance problems seen in a distributed storage system are presented, and Spectroscope’s primary assumptions and algorithms are evaluated