USENIX Association 11th USENIX Conference on File and Storage Technologies (FAST ’13) 229 To Zip or not to Zip: Effective Resource Usage for Real-Time Compression∗

Real-time compression for primary storage is quickly becoming widespread as data continues to grow exponen-tially, but adding compression on the data path consumes scarce CPU and memory resources on the storage sys-tem. Our work aims to mitigate this cost by introducing methods to quickly and accurately identify the data that will yield significant space savings when compressed. The first level of filtering that we employ is at the data set level (e.g., volume or file system), where we estimate the overall compressibility of the data at rest. Accord-ing to the outcome, we may choose to enable or disable compression for the entire data set, or to employ a sec-ond level of finer-grained filtering. The second filtering scheme examines data being written to the storage system in an online manner and determines its compressibility. The first-level filtering runs in mere minutes while providing mathematically proven guarantees on its esti-mates. In addition to aiding in selecting which volumes to compress, it has been released as a public tool, allow-ing potential customers to determine the effectiveness of compression on their data and to aid in capacity plan-ning. The second-level filtering has shown significant CPU savings (up to 35%) while maintaining compression savings (within 2%).