Design of an RDMA communication middleware for asynchronous shuffling in analytical processing

A key component in a distributed parallel analytical processing engine is shuffling, the distribution of data to multiple nodes such that the computation can be done in parallel. In this paper we describe the initial design of a communication middleware to support asynchronous shuffling of data among multiple processes on a distributed memory environment. The proposed middleware relies on RDMA (Remote Direct Memory Access) operations to transfer data, and provides basic operations to send and queue data on remote machines, and to retrieve this queued data. Preliminary results show that the RDMA-based middleware can provide a 75% reduction on communication costs, when compared with a traditional sockets implementation.This work has received funding from the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619606, project LeanBigData Ultra-Scalable and Ultra-Efficient Integrated and Visual Big Data Analytics (http://leanbigdata.eu).