Performance evaluation of infiniband with pci express

Abstract — In this paper, we present an initial performance evaluation of InfiniBand HCAs from Mellanox with PCI Express interfaces. We compare the performance with HCAs using PCI-X interfaces. Our results show that InfiniBand HCAs with PCI Express can achieve significant performance benefits. Compared with HCAs using 64 bit/133 MHz PCI-X interfaces, they can achieve 20%–30 % lower latency for small messages. The small message latency achieved with PCI Express is around 3.8 s, compared with the 5.0 s with PCI-X. For large messages, HCAs with PCI Express using a single port can deliver unidirectional bandwidth up to 968 MB/s and bidirectional bandwidth up to 1916 MB/s, which are 1.24 and 2.02 times of the peak bandwidths achieved by HCAs with PCI-X, respectively. When both the ports of the HCAs are activated, HCAs with PCI Express can deliver a peak unidirectional bandwidth of 1486 MB/s and aggregate bidirectional bandwidth up to 2729 MB/s, which are 1.93 and 2.88 times of the peak bandwidths obtained using HCAs with PCI-X. PCI Express also improves performance at the MPI level. A latency of 4.6 s with PCI Express is achieved for small messages. And for large messages, uni-directional bandwidth of 1497 MB/s and bi-directional bandwidth of 2724 MB/s are observed. I