MT-MPI: Multithreaded MPI for many-core environments

Many-core architectures, such as the Intel Xeon Phi, provide dozens of cores and hundreds of hardware threads. To uti-lize such architectures, application programmers are increas-ingly looking at hybrid programming models, where multi-ple threads interact with the MPI library (frequently called “MPI+X ” models). A common mode of operation for such applications uses multiple threads to parallelize the compu-tation, while one of the threads also issues MPI operations (i.e., MPI FUNNELED or SERIALIZED thread-safety mode). In MPI+OpenMP applications, this is achieved, for example, by placing MPI calls in OpenMP critical sections or outside the OpenMP parallel regions. However, such a model often means that the OpenMP threads are active only during the parallel computation phase and idle during the MPI calls, resulting in wasted computational resources. In this paper, we present MT-MPI, an internally multithreaded MPI imple-mentation that transparently coordinates with the threading runtime system to share idle threads with the application. It is designed in the context of OpenMP and requires modifi-cations to both the MPI implementation and the OpenMP runtime in order to share appropriate information between them. We demonstrate the benefit of such internal paral-lelism for various aspects of MPI processing, including de-rived datatype communication, shared-memory communica-tion, and network I/O operations