Predictive Assessment of Model Uncertainty

In this dissertation, we use several predictive approaches to address the problem of model selection uncertainty. Unless a data generator is known a priori, there is uncertainty associated with model selection in any statistical analysis. We present three related, but different predictive methods that handle model uncertainty in dif- ferent ways. The first approach uses predictive stability in response to perturbing the data to choose a robust model from a pre-specified list. Here we focus on shrinkage methods used in a high dimensional regression setting, but note that our predictive stability criteria can be used for generic lists of pre-specified models. The second approach uses model averaging techniques to obtain a predictive distribution that accounts for all sources of uncertainty stemming from modeling choices made by an analyst. Our main contribution comes from defining a finite dimensional discrete random variable to decompose the predictive variance into portions representing com- ponents of the modeling procedure. The terms in the decomposition are analogous to the terms in Cochran’s theorem decomposition used in ANOVA. Each term in the decomposition is distributed like a weighted chi-square, but exact distributions are not available. Hence, we also present a bootstrap testing procedure that allows us to to determine if a component of the modeling strategy significantly contributes to the predictive variance. The third approach gives asymptotically valid predictive intervals for GLM’s, as well a small sample cross validation and bootstrap predictive intervals when asymptotic results cannot be assumed