InfiniBand RDMA over PCI Express Networks

Remote Direct Memory Access (RDMA) is a data transmission technology featuring high bandwidth, low latency, and low computational overhead. RDMA can be used to transmit data between applications running on interconnected computers while bypassing the kernel network stack. The efficiency of bypassing the kernel makes RDMA ideal for High-Performance Computing (HPC) and cloud computing. RDMA works by offloading network transfers in the kernel to network adapters with RDMA capabilities. These network adapters can read and write the virtual memory of an application with Direct Memory Access (DMA). DMA operations are executed safely with a virtual-to-physical memory map created through RDMA semantics. RDMA capable network adapters exist for InfiniBand and Ethernet network fabrics, and transport agnostic RDMA APIs enable applications to run on either fabric. In this thesis, we have developed an RDMA transport named "RDMA over PCIe (RoPCIe)," intended to be used in computer clusters interconnected with PCIe Non-Transparent Bridges (NTB). Inspired by RDMA over Converged Ethernet (RoCE), this transport takes advantage of established RDMA APIs to provide a new mode of transport for existing RDMA applications. Bringing a technology developed primarily for InfiniBand to PCIe interconnect, an esoteric transmission technology compared to InfiniBand, is one of the primary motivations for RoPCIe. We have implemented the RoPCIe transport for Linux and made it available to applications through RDMA APIs in both kernel-space and user-space. The primary component of the RoPCIe implementation is a virtual RDMA device, registered with the kernel RDMA subsystem by a virtual device driver. This virtual device acts as an intermediary between the RDMA API and the NTB device driver, translating RDMA operations into NTB operations. To add support for RoPCIe transport in user-space, we have implemented a dynamically linked plugin for a library called "Libibverbs". The result of our RoPCIe implementation is a working proof-of-concept RDMA transport. RoPCIe can run unmodified RDMA applications over PCIe interconnect and achieves performance comparable to InfiniBand transport