Benefits of High Speed Interconnects to Cluster File Systems: A Case Study with Lustre

Cluster file systems and Storage Area Networks (SAN) make use of network IO to achieve higher IO bandwidth. Effective integration of networking mech-anisms is important to their performance. In this paper, we perform an evaluation of a popular cluster file sys-tem, Lustre, over two of the leading high speed cluster interconnects: InfiniBand and Quadrics. Our evalua-tion is performed with both sequential IO and parallel IO benchmarks in order to explore the capacity of Lustre under different communication characteristics. Experi-mental results show that direct implementations of Lus-tre over both interconnects can improve its performance, compared to an IP emulation over InfiniBand (IPoIB). The performance of Lustre over Quadrics is compara-ble to that of Lustre over InfiniBand with the platforms we have. Latest InfiniBand products can embrace latest technologies, such as PCI-Express and DDR, and pro-vide higher capacity. Our results show that over a Lustre file system with two Object Storage Servers (OSSs), In-finiBand with PCI-Express technology can improve Lus-tre write performance by 24%. Furthermore, our experi-mental results indicate that Lustre meta-data operations do not scale with an increasing number of OSSs, in spite of using high performance interconnects