P-Socket: Optimizing a communication library for a PCIe-based intra-rack interconnect

Data centers require efficient, low-cost, exible interconnects to manage the rapidly growing internal traffic generated by an increasingly diverse set of applications. To meet these re-quirements, data center networks are increasingly employing alternatives such as RapidIO, Freedom, and PCIe, which re-quire fewer physical devices and/or have simpler protocols than more traditional interconnects. These networks offer raw high performance communication capabilities, but sim-ply using them for conventional TCP/IP-based communica-tion fails to realize the potential performance of the physical network. Here we analyze causes for this performance loss for the TCP/IP protocol over one such fabric, PCIe, and we explore a hardware/software solution that mitigates over-heads and exploits PCIe's advanced features. The result is P-Socket, an efficient library that enables legacy socket applications to run without modification. Our experiments show that P-Socket achieves an end-To-end latency of 1.2s and effiective bandwidth of up to 2.87GB/s (out of a theo-retical peak of 3.05GB/s)