lprof: A Non-intrusive Request Flow Profiler for Distributed Systems

Applications implementing cloud services, such as HDFS, Hadoop YARN, Cassandra, and HBase, are mostly built as distributed systems designed to scale. In order to analyze and debug the performance of these systems efficiently, it is essential to understand the performance behavior of system requests, both in aggregate and individually. lprof is a profiling tool that automatically reconstructs the execution flow of each request in a distributed application. It infers the execution-flow from logs and binary code and thus does not require any modifications to the application. lprof first statically analyzes an application’s binary code to infer how logs can be parsed so that the dispersed and intertwined log entries can be stitched and associated to specific individual requests. We evaluate lprof on the four widely used distributed services mentioned above. The result shows that lprof ’s precision in request extraction is 90%, and lprof is helpful in diagnosing 65% of the random sampled real-world performance anomalies.M.A.S