Performance analysis and evaluation of PCIe 2.0 and quad-data rate InfiniBand

High-performance systems are undergoing a major shift as commodity multi-core systems become increas-ingly prevalent. As the number of processes per com-pute node increase, the other parts of the system must also scale appropriately to maintain a balanced sys-tem. In the area of high-performance computing, one very important element of the overall system is the net-work interconnect that connects compute nodes in the system. InfiniBand is a popular interconnect for high-performance clusters. Unfortunately, due to limited bandwidth of the PCI-Express fabric, InfiniBand perfor-mance has remained limited. PCI-Express (PCIe) 2.0 has recently become available and has doubled the transfer rates available. This ad-ditional I/O bandwidth balances the system and makes higher data rates for external interconnects such as In-finiBand feasible. As a result, InfiniBand Quad-Data Rate (QDR) mode has become available on the newest Mellanox InfiniBand Host Channel Adapter (HCA) with a 40 Gb/sec signaling rate. In this paper we perform an in-depth performance analysis of PCIe 2.0 and the effect of increased InfiniBand signaling rates. We show that even using the Double Data Rate (DDR) interface, PCIe 2.0 enables a 25 % improvement in NAS Parallel Bench-mark IS performance. Furthermore, we show that when using QDR on PCIe 2.0, network loopback can outper-form a shared memory message passing implementation. We show that increased interconnection bandwidth sig-nificantly improves the overall system balance by lower-∗This research is supported in part by Department of En-ergy grants #DE-FC02-06ER25749 and #DE-FC02-06ER25755; NSF grants #CNS-0403342 and #CCF-0702675; grants from Intel, Mel