A Two Part Study in the Evaluation of Quantitative Precipitation Forecasts from Convection-Allowing Models

Previous studies examining convection-allowing models (CAMs), as well as NOAA/Hazardous Weather Testbed Spring Forecasting Experiments (SFEs) have typically emphasized “day 1” (12-36 h) forecast guidance. These studies find a distinct advantage in CAMs relative to models that parameterize convection, especially for fields strongly tied to convection like precipitation. During the 2014 SFE, “day 2” (36-60 h) forecast products from a CAM ensemble provided by the Center for Analysis and Prediction of Storms at the University of Oklahoma were examined. Quantitative Precipitation Forecasts (QPFs) from the CAPS ensemble, known as the Storm Scale Ensemble Forecast (SSEF) system, are compared to NCEP’s operational Short Range Ensemble Forecast (SREF) system, which provides lateral boundary conditions for the SSEF, to see if the CAM ensemble outperforms the SREF through forecast hours 36-60. Equitable Threat Score (ETS) was computed for precipitation thresholds ranging from 0.10 in. to 0.75 in. for each SSEF and SREF member, as well as ensemble means, for 3 h accumulation periods. The ETS difference between the SSEF and SREF peaked during hours 36-42. Probabilistic forecasts were evaluated using the area under the receiver operating characteristic curve (ROC Area). The SSEF had higher values of ROC Area, especially at thresholds ≥ 0.50 in. Additionally, time-longitude diagrams of diurnally averaged rainfall were constructed for each SSEF/SREF ensemble member. Spatial correlation coefficients between forecasts and observations in time-longitude space indicated that the SSEF depicted the diurnal cycle much better than the SREF, which under-forecasted precipitation with a peak that had a 3 h phase lag. A minority of SREF members performed well. One component of the 2016 NOAA/Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) examined the impact of radar data assimilation on convection-allowing model (CAM) ensemble forecasts using two similarly configured 10-member ensembles with (rad-ens) and without (norad-ens) radar data assimilation, which were provided by the Center of Analysis and Prediction of Storms and the National Severe Storms Laboratory, respectively. While previous works have quantified the impact of radar data assimilation on the skill of deterministic forecasts, the impact on ensemble forecast skill has yet to be examined. Equitable threat scores (ETSs) were computed for precipitation thresholds ranging from 0.10 to 0.75 in., and neighborhood-based ETS (ETSr) was computed for radii ranging from 8- to 60-km. The ETS difference between rad-ens and norad-ens peaked at forecast hour 3, but existed until forecast hours 12-15. As the radius in the ETSr increased, the difference in ETSr between the rad-ens and norad-ens increased, even out to hour 36. 3-h probabilistic QPFs were evaluated using the area under the ROC curve (ROC Area). Similar to ETS, ROC area results showed a difference in favor of the rad-ens out to forecast hour 15. The rad-ens had slightly greater variance and lower MSE out to hour 15, then both ensembles had nearly identical variance/MSE values from hours 15 to 36. Rank histograms were similar for each ensemble, and indicated over-forecasting. Most of the subjective evaluations filled out by 2016 SFE participants indicated that the positive effects of data assimilation lasted less than 12 h. A small minority of the respondents saw an advantage at lead times of 15-24 h