Report from study of data compression assessment (D2.7)

An increase in processing power enabled to increase resolution and the number of ensemble members for both weather and climate simulations during the last decades. As a result, the amount of model input and output data that needs to be stored and processed increased significantly. However, supercomputing storage capacities did not grow at the same rate as processing power. Today, data storage and data processing generates significant cost for weather and climate modelling centres. This deliverable suggests three new compression methods to reduce data volume for model output of ensemble simulations. If the new methods are used, the number of bits that is used to store individual variables can be reduced to reduce overall data volume while keeping the loss of information minimal. Two of the new methods rely on similarities between ensemble members in ensemble forecasts. If these methods are used, numerical precision will be high when ensemble spread is small and precision will decrease with increasing ensemble spread throughout the forecast. The methods are tested for model output of operational ensemble forecasts at ECMWF.ESiWACE has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 67519