Exploring Data Compression in Distributed File Systems

Abstract—The ever-growing imbalance between computation and I/O is one of the fundamental challenges for current petascale and future exascale systems. As these large systems become more compute heavy by design, it is becoming harder to support dataintensive applications with adequate performance. To narrow the gap between computation and I/O, parallel and distributed file systems have been introduced. A popular approach to further ameliorate the I/O pressure is data compression, which aims to reduce the data size and stress on the network. However, it unsurprisingly brings new challenges such as computational overhead, programmability, and co-design with other components in the orchestra of large systems. In order to reduce the computation overhead of data compression in these large scale systems, this paper proposes a new compression mechanism, namely multireference compression (MRC), by introducing multiple references to the underlying physical chunks. These multiple references across the physical chunks enable an efficient decompression of requested chunk subsets, thus bypassing the overhead on decompressing the data outside of the requested range. To make MRC easy to use by scientific computing applications, we have adopted the FUSE kernel module to support POSIX. The MRC system can be used in conjunction with parallel file systems (e.g. GPFS) with no modifications needed to either the application or the high level I/O libraries (e.g. netCDF, HDF5). Towards achieving even higher I/O performance, we have proposed and executed a tight integration of MRC with the FusionFS distributed file system, and explored MRC’s impact on basic file system components such as metadata management and data movement. We show how MRC is able to improve parallel and distributed file system performance by up to 2X over these file systems without compression. Keywords-data compression; parallel file systems; distributed file systems I