Experiences with Fast Forwarding on Myrinet

As cluster services become increasingly complex, aware-ness of the capabilities of those services are pushed back into the network, and service aware networking components are now common. Traffic entering and exiting cluster ser-vices is likely to be forwarded through a front-end as the performance and economic benefits of resource-allocation and load-balancing across the cluster service are realized. Packet routing, an end-to-end distributed service, is diverg-ing from packet forwarding, a localized service specific task, as web serving, caching, redirection, and other gen-eral front-end host-based architectures increase in popular-ity. This paper introduces payload caching, a technique that caches incoming data payloads on the network interface, transparently reducing the number of I/O transfers needed to forward a packet. With payload caching, data is still transferred to host memory for intelligent redirection or host processing. Unmodified data remains cached on the network interface card(NIC) where it may be sent without crossing the I/O interconnect again. We report user-level results for three generations of LANai processor architec-ture and a queuing model showing that current NIC mem-ory sizes are sufficiently large to get a 100 % payload cache hit rate under certain conditions. Our prototype on the next generation Myrinet adapter (LANai-9) delivers an 89% improvement in forwarding bandwidth on a 32-bit 33Mhz PCI system, in this case bandwidth for a single 4KB packet stream goes from 56.67MB/s without payload caching t