Analyzing Message Passing Programs on the Cray T3E with PAT and VAMPIR

Writing efficient parallel programs for a massively parallel system like the Cray T3E is still a difficult task because such programs are typically very large and complex and their dynamic behavior is difficult to understand or predict. Therefore, runtime performance analysis tools are needed on such systems in addition to the normal programming environment tools like editors and debuggers.For the Cray T3E, Cray Research implemented and provides two performance analysis tools, Apprentice and PAT. Apprentice is a profiling tool which uses source code instrumentation through compiler switches and provides statistics on the level of functions and basic blocks. PAT, the Performance Analysis Tool, is actually several tools in one. It provides profiling through sampling and access to hardware performance information. It also includes an object code instrumentor which can be used for detailed call site profiling and function level hardware performance statistics.In a collaboration between Cray Research and Forschungszentrum Jülich, PAT was extended to also support event tracing. In this paper, we describe how the new extended PAT and VAMPIR, an event trace browser developed by Forschungszentrum Jülich, can be used to analyze message passing programs on the Cray T3E. The powerful trace browsing features of VAMPIR make it a perfect extension to PAT's object instrumentation and tracing functionality.First, the features of PAT are described in detail. In order to analyze message passing programs, the message passing libraries of the Cray T3E (MPI, PVM, and SHMEM) needed to be instrumented. This feature is described next. We then give an overview of VAMPIR and its functionality. With two small examples, we show how the combination of PAT's object instrumentation features, the new message passing function wrapper library, and VAMPIR's trace displays can be used to analyze message passing programs on the Cray T3E to any detail