The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems

Today's distributed tracing frameworks are ill-equipped to troubleshoot rareedge-case requests. The crux of the problem is a trade-off between specificityand overhead. On the one hand, frameworks can indiscriminately select requeststo trace when they enter the system (head sampling), but this is unlikely tocapture a relevant edge-case trace because the framework cannot know whichrequests will be problematic until after-the-fact. On the other hand,frameworks can trace everything and later keep only the interesting edge-casetraces (tail sampling), but this has high overheads on the traced applicationand enormous data ingestion costs. In this paper we circumvent this trade-off for any edge-case with symptomsthat can be programmatically detected, such as high tail latency, errors, andbottlenecked queues. We propose a lightweight and always-on distributed tracingsystem, Hindsight, which implements a retroactive sampling abstraction: insteadof eagerly ingesting and processing traces, Hindsight lazily retrieves tracedata only after symptoms of a problem are detected. Hindsight is analogous to acar dash-cam that, upon detecting a sudden jolt in momentum, persists the lasthour of footage. Developers using Hindsight receive the exact edge-case tracesthey desire without undue overhead or dependence on luck. Our evaluation showsthat Hindsight scales to millions of requests per second, adds nanosecond-leveloverhead to generate trace data, handles GB/s of data per node, transparentlyintegrates with existing distributed tracing systems, and successfully persistsfull, detailed traces in real-world use cases when edge-case problems aredetected.