CrossWalk: A Tool for Performance Proling Across the User-Kernel Boundary (ParCo Preprint)

Performance proling of application programs is often a challenging task. One problem in the analysis is that applications often spend signicant amount of their time in the operating system. As a result, conventional user-level prolers can, at best, trace the performance bottleneck to a particular system call. This information is often insucient for the programmer to deal with the performance problem|system calls can be quite complex, making it hard to pinpoint the cause of the problem. To address this deciency, we have designed a tool called CrossWalk that is able to do performance analysis across the kernel boundary. CrossWalk starts proling at the user level, proles the main function, then its callees and walks further down the application call graph, rening the performance problem to a particular function. If it determines that this function is a system call, it walks into the kernel code and starts traversing the kernel call graph until it locates the ultimate bottleneck. The key technologies in CrossWalk are dynamic program instrumentation and dynamic kernel instrumen-tation. For the former, we use an existing library called Dyninst API. For the latter, we have designed a new framework called Kerninst API with an interface modeled after Dyninst. When combined, the two libraries provide a unied and powerful interface for building cross-boundary tools. We demonstrate the usefulness of the cross-boundary approach by analyzing the Squid proxy server with CrossWalk. By drilling down into the kernel, we were able to identify the ultimate cause of its performance problems and remove them by modifying the application's source code.