Rack-Scale Storage Fabric: A Practical Way to Build Best-Fit Infrastructure for High-Performance Data Processing

This paper is to address the resource utilization problem for high-performance data processing applications in a large IDC (Internet Data Center) environment. On one hand, each application calls for a best-fit infrastructure with a specific compute-storage ratio, to achieve the highest resource utilization while meeting its performance requirement. And such a ratio varies among applications. On the other hand, IDCs have always been trying to unify the infrastructures for lower TCO (Total Cost of Ownership). Therefore, it’s getting harder and harder to adapt infrastructures to application needs. This issue results in significant waste of investment in large IDCs. Furthermore, the high-performance data processing applications always require the infrastructure to offer as high compute-storage performance as a DAS (Direct Attached Storage) server, which remains as a great challenge when addressing the resource utilization problem. This paper, as part of Baidu-Intel joint research program, first evaluates the state-of-the-art solutions, and then introduces a more practical infrastructure, the core of which is rack-scale storage fabric. This infrastructure disaggregates compute units and storage units by a SAS (Serial Attached SCSI) fabric, and allows to compose logical servers with arbitrary computer-storage ratios within a rack. And the experiments in Baidu’s research environment show that the logical servers exhibit the similar throughput/IOPS as DAS servers, and also their compute-storage ratios can best-fit the needs of different Hadoop applications