Design and Implementation of a Network-Attached Accelerator to Improve Data Movement in HPC Environments

In this work, the concept of a network-attached accelerator was developed. This novel node type connects accelerator and network interface card directly via an artificial PCIe interface. By exploiting the additional functions of the EXTOLL network card, this remote PCIe hierarchy can be integrated into any system with an EXTOLL network interface card via the network without requiring any modifications to the accelerator driver. A Intel R Xeon PhiTMwas used which can be successfully booted by the remote PCIe hierarchy over the network and is then available as a standalone node. In addition, the Intel R Xeon PhiTMitself can actively communicate with other Intel R Xeon PhiTMwithout the need for a CPU. The accelerator and network card form a single unit that can communicate directly with others via the highly specialized EXTOLL network, forming an cluster of accelerators. Various experiments demonstrate, that this combination has a higher bandwidth and a lower latency than common systems. Several prototype systems have been developed to evaluate the concept and reduce the packing density by a factor of four. This is the most densely packed HPC system to date. The enormous cooling demand could only be met by using a novel cooling medium NovecTM 649 . An developed 2-phase cooling system can cool the 12 kW power dissipation of the 32 network-attached accelerators with a total of 32 TFLOP of computing power only by the power consumption of a single circulating pump. As a result, the GreenICE system achieves an unprecedented power usage effectiveness of 1.01