Efficient, unified, and scalable performance monitoring for multiprocessor operating systems

Programming, understanding, and tuning the performance of large multiprocessor systems is challenging. Experts have difficulty achieving good utilization for applications on large machines. The task of implementing a scal-able system such as an operating system or database on large machines is even more challenging. And the impor-tance of achieving good performance on multiprocessor machines is increasing as the number of cores per chip in-creases and as the size of multiprocessors increases. Cru-cial to achieving good performance is being able to under-stand the behavior of the system. We have developed an efficient, unified, and scalable tracing infrastructure that allows for correctness debug-ging, performance debugging, and performance monitor-ing of an operating system. The infrastructure allows variable-length events to be logged without locking and provides random access to the event stream. The infras-tructure allows cheap and parallel logging of events by applications, libraries, servers, and the kernel. The in-frastructure was designed for K42, a new open-source re-search kernel designed to scale near perfectly on large cache-coherent 64-bit multiprocessor systems. The tech-niques are generally applicable, and many of them have been integrated into the Linux Trace Toolkit. In this paper, we describe the implementation of the infrastructure, how we used the facility, e.g., analyzing lock contention, to un-derstand and achieve K42’s scalable performance, and the lessons we learned. The infrastructure has been invaluable to achieving great scalability