Optimizing the TCP stack in Linux for transparent and efficient use of RDMA over PCI Express

Remote Direct Memory Access (RDMA) allows for direct memory access between separate computers and can decrease the overhead of inter-process communications (IPC) between connected computers. In combination with PCI Express (PCIe) hardware, RDMA can offer high bandwidth, low latency, and lossless communication. It also empowers the system to make better use of system resources through smart memory sharing, CPU bypass, and removing overhead needed to ensure successful data transfer. To use RDMA, the application software needs to be adapted to the specific RDMA hardware and libraries. We propose to implement a tailored TCP stack, Local TCP (LTCP), to allow RDMA communication using standard TCP sockets. This will give applications easy access to the high performance RDMA over PCIe provides. This thesis looks at how we can create a transport protocol in Linux that is more optimized for RDMA over PCIe. It also explores how to make transparent use of LTCP through the standard TCP socket interface. We have implemented a kernel module to add a minimal transport protocol for local RDMA communication on Linux, named Local TCP (LTCP). The module provides a proof-of-concept for how regular TCP transmissions can be automatically remapped to utilize another protocol for specific IP addresses. The remap allows for transparent use of the LTCP protocol, which implements a small tailored network stack. The LTCP protocol implements a new approach for transferring data using RDMA, copying data directly from user-space buffers on one machine to the user-space buffers on another machine. This solution removes the need for intermediate kernel buffers and handles the direct copy-operation through DMA. LTCP does this completely transparent to the user-space programs using TCP. Our benchmarks of the LTCP implementation shows improved performance in process-to-process communication compared to the other directly comparable solutions in the benchmark. The benchmark covers performance in terms of bandwidth, latency, CPU usage and memory usage