Virtualized Multi-Channel RDMA with Software-Defined Scheduling

In HPC, low latency communication between remote processes is crucial to application performance. InfiniBand networks can reduce the latency but require special and costly network interface cards, which are loosely coupled with CPU, thus needing to constantly re-pin pages. In this work, we virtualize the hardware Direct Memory Access (DMA) engine of the MPSoC in software by using an ARM Cortex-R5 to provide 1024 virtual DMA channels for the processes and to prioritize the small critical transfers, reducing the latency by a factor of 10x. Furthermore, we leverage the System Memory Management Unit (SMMU) available in the Zynq Ultrascale+ MPSoC’s to allow user based nodes, achieving a 6x latency improvement over kernel based solutions