The importance of non-data-communication overheads in mpi

With processor speeds no longer doubling every 18–24 months owing to the exponential increase in power con-sumption and heat dissipation, modern high-end comput-ing systems tend to rely less on the performance of single processing units and instead rely on achieving high per-formance by using the parallelism of a massive number of low-frequency/low-power processing cores. Using such low-frequency cores, however, puts a premium on end-host pre- and post-communication processing required within communication stacks, such as the Message Passing Inter-face (MPI) implementation. Similarly, small amounts of serialization within the communication stack that were acceptable on small/medium systems can be brutal on massively parallel systems. Thus, in this paper, we study the different non-data-communication overheads within the MPI implementation on the IBM Blue Gene/P system. Specifically, we analyze various aspects of MPI, including the MPI stack overhead itself, overhead of allocating and queueing requests, queue searches within the MPI stack, multi-request operations, and various others. Our experi-ments, that scale up to 131,072 cores of the largest Blue Gene/P system in the world (80 % of the total system size), reveal several insights into overheads in the MPI stack, which were not previously considered significant, but can have a substantial impact on such massive systems. Key words: Blue Gene/P, MPI, non-data-communication overheads