Advanced Compression Algorithms for netCDF Datasets with the Community Codec Repository

For decades, the dominant geoscience and engineering array-oriented data storage format, netCDF/HDF, supported only one open source lossless compression algorithm, DEFLATE. We introduce the Community Codec Repository (CCR), a shared library framework that enables netCDF/HDF datasets and applications to transparently employ modern open source compression methods. These methods accelerate compression/decompression and improve compression ratios, which thereby reduce storage requirements, power consumption, and greenhouse emissions. Four CCR codecs have been integrated directly into the netCDF library for widespread use. These include two project-developed lossy codecs that quantize data 20-40% more efficiently than before. Geoscience and engineering datasets (and archives like CMIP7), can now store data in half the amount of space required with DEFLATE while still preserving all scientifically meaningful information.NSF project OAC-200499