Improved Earth System Prediction Using Large Ensembles and Machine Learning

The purpose of this thesis is to examine and advance North American weatherpredictability from weather to subseasonal time-scales. Specifically, it focuses on 1) developing machine learning/deep learning methods and models to improve predictability through numerical weather prediction (NWP) post-processing on weather time-scales (0-7 days) and 2) examining the physical mechanisms which govern the evolution of the predictable components and noise components of teleconnection modes on subseasonal time-scales (7 days - 1 month). NWP deficiencies (e.g., sub-grid parameterization approximations), nonlinear error growth associated with the chaotic nature of the atmosphere, and initial condition uncertainty lead initial small forecast errors to eventually result in weather predictions which are as skillful as random forecasts. A portion of these forecast errors are inherent to the NWP models alone, systematic biases. The first two chapters develop cutting-edge vision-based deep-learning algorithms to advance the current state-of-the-art NWP post-processing and correct these systematic biases. Using dynamic forecasts of North Pacific integrated vapor transport (IVT) as a test case, we develop post-processing systems which are spatially aware, readily encode non-linear predictor interaction, easily ingest ancillary weather variables, and have state of the art training methods that systematically prevent model overfitting. Further, we outline a framework to quantify uncertainty in single-point (deterministic) forecasts using neural networks. The uncertainty is shown to be probabilistically rigorous, leading to calibrated probabilistic forecasts which outperform or compete with calibrated dynamic NWP ensemble systems for IVT under atmospheric river conditions.The second half of this thesis shifts focus to subseasonal time scales and examinespredictability in the Pacific North American (PNA) sector in boreal winter. Particularly, it investigates the physical mechanisms involved in the intraseasonal modulation of atmospheric Signal-to-Noise (SN), and how it is affected by slowly varying climate modes (ENSO and MJO). These mechanisms are further explored using a fully-coupled hindcast of the 20th century, showing that the increased SN leads to high model forecast skill at subseasonal timescales in particular forecast windows of opportunity. Additionally, we reveal the MJO as the largest growing mode of tropical forecast uncertainty which directly influences PNA forecast certainty