Beyond the CPU: Hardware Performance Counter Monitoring on Blue Gene/Q

Abstract. The Blue Gene/Q (BG/Q) system is the third generation in the IBM Blue Gene line of massively parallel, energy efficient supercom-puters that increases not only in size but also in complexity compared to its Blue Gene predecessors. Consequently, gaining insight into the intricate ways in which software and hardware are interacting requires richer and more capable performance analysis methods in order to be able to improve efficiency and scalability of applications that utilize this advanced system. The BG/Q predecessor, Blue Gene/P, suffered from incompletely im-plemented hardware performance monitoring tools. To address these limitations, an industry/academic collaboration was established early in BG/Q’s development cycle to insure the delivery of effective performance tools at the machine’s introduction. An extensive effort has been made to extend the Performance API (PAPI) to support hardware performance monitoring for the BG/Q platform. This paper provides detailed infor-mation about five recently added PAPI components that allow hardware performance counter monitoring of the 5D-Torus network, the I/O sys-tem and the Compute Node Kernel in addition to the processing cores on BG/Q. Furthermore, we explore the impact of node mappings on the perfor-mance of a parallel 3D-FFT kernel and use the new PAPI network com-ponent to collect hardware performance counter data on the 5D-Torus network. As a result, the network counters detected a large amount of redundant inter-node communications, which we were able to completely eliminate with the use of a customized node mapping.