Using Visualizations to Monitor Changes and Harvest Insights from a Global-Scale Logging Infrastructure at Twitter

Fig. 1. Scribe Radar (Icicle View) — This interactive visualization displays a collection of log events from a hypothetical product that operates on three platforms: strawberry, coconut, and banana. Log events follow a six-level naming hierarchy (client:page:section:component:element:action). Here, we see that the event strawberry:search:personal:-:-:impression in-creased in frequency compared to seven days ago, indicated by a light blue rectangle, while a light red rectangle shows that the event strawberry:inbox:inbox:conversation:-:impression dropped slightly. Abstract — Logging user activities is essential to data analysis for internet products and services. Twitter has built a unified logging infrastructure that captures user activities across all clients it owns, making it one of the largest datasets in the organization. This paper describes challenges and opportunities in applying information visualization to log analysis at this massive scale, and shows how various visualization techniques can be adapted to help data scientists extract insights. In particular, we focus on two scenarios: (1) monitoring and exploring a large collection of log events, and (2) performing visual funnel analysis on log data with tens of thousands of event types. Two interactive visualizations were developed for these purposes: we discuss design choices and the implementation of these systems, along with case studies of how they are being used in day-to-day operations at Twitter