Using SCTP to hide latency in MPI programs

A difficulty in using heterogeneous collections of ge-ographically distributed machines across wide area net-works for parallel computing is the huge variability in message latency that is orders of magnitude larger than parallel programs executing on dedicated systems. This variability is in part due to the underlying network bandwidth and latency which can vary dramatically ac-cording to network conditions. Although such an envi-ronment is not suitable for many message passing pro-grams there are those programs that can take advantage of it. Using SCTP (Stream Control Transmission Proto-col) for MPI, we show how to reduce the effect of la-tency on task farm programs to allow them to effec-tively execute in high latency environments. SCTP is a recently standardized transport level protocol that has a number of features that make it well-suited to MPI and our goal is to reduce the effect of latency on MPI programs in wide area networks. We take advantage of SCTP’s improved congestion control as well as its ability to have multiple independent message streams over a single connection to eliminate the head of line blocking that can occur in TCP-based middleware. The use of streams required a novel use of MPI tags to identify independent streams rather than dif-ferent types of messages. We describe the design of a task farm template that exploits streams, uses buffer-ing and pipelining of task requests to improve its per-formance under network loss and variable latency. We use these techniques to improve the performance of two real-world MPI programs: a robust correlation matrix computation and mpiBLAST.