Performance Evaluation of RDMA over IP: A Case Study with the Ammasso Gigabit Ethernet NIC

Remote Direct Memory Access (RDMA) has been proposed to overcome the limitations of traditional send/receive based communication protocols like sockets. The immense potential of RDMA to improve the communication performance while being extremely conservative on resource requirements has made RDMA the most sought after feature in current and next generation networks. Recently, there are many active efforts to enable RDMA over IP networks and fabrication of RDMA-enabled Ethernet NICs. However, the performance of RDMA over IP networks has not been quantitatively evaluated over WAN environments yet. In this paper, we evaluate the performance of RDMA over IP networks with the Ammasso Gigabit Ethernet NIC while emulating high delay WANs and varying load on remote node. We observe that RDMA is beneficial especially under heavy load conditions. More importantly, even with a high delay, RDMA can provide better communication progress and requires less CPU resources as compared to the traditional sockets over TCP/IP. Further, we show that RDMA can support high performance intra-cluster communication while providing a unified communication interface for inter- and intra-cluster communication. To the best of our knowledge, this is the first quantitative study of RDMA over IP on a WAN setup