MPI+ threads: Runtime contention and remedies

Hybrid MPI+Threads programming has emerged as an alternative model to the “MPI everywhere ” model to better handle the increas-ing core density in cluster nodes. While the MPI standard allows multithreaded concurrent communication, such flexibility comes with the cost of maintaining thread safety within the MPI imple-mentation, typically implemented using critical sections. In contrast to previous works that studied the importance of critical-section granularity in MPI implementations, in this paper we investigate the implication of critical-section arbitration on communication per-formance. We first analyze the MPI runtime when multithreaded concurrent communication takes place on hierarchical memory sys-tems. Our results indicate that the mutex-based approach that most MPI implementations use today can incur performance penalties due to unfair arbitration. We then present methods to mitigate these penalties with a first-come, first-served arbitration and a priority locking scheme that favors threads doing useful work. Through eval-uations using several benchmarks and applications, we demonstrate up to 5-fold improvement in performance