On Bayesian Regression Regularization Methods

Regression regularization methods are drawing increasing attention from statisticians for more frequent appearance of high-dimensional problems. Regression regularization achieves simultaneous parameter estimation and variable selection by penalizing the model parameters. In the first part of this thesis, we focus on the elastic net [73], which is a flexible regularization and variable selection method that uses a mixture of L1 and L2 penalties. It is particularly useful when there are much more predictors than the sample size. We proposes a Bayesian method to solve the elastic net model using a Gibbs sampler. While the marginal posterior mode of the regression coefficients is equivalent to estimates given by the non-Bayesian elastic net, the Bayesian elastic net has two major advantages. Firstly, as a Bayesian method, the distributional results on the estimates are straightforward, making the statistical inference easier. Secondly, it chooses the two penalty parameters simultaneously, avoiding the “double shrinkage problem” in the elastic net method. Real data examples and simulation studies show that the Bayesian elastic net behaves comparably in prediction accuracy but performs better in variable selection. The second part of this thesis investigates Bayesian regularization in quantile regression. Quantile regression is a method that models the relationship between the response variable and covariates through the population quantiles of the response variable. By proposing a hierarchical model framework, we give a generic treatment to a set of regularization approaches, including lasso, elastic net and group lasso. Gibbs samplers are derived for all cases. This is the first work to discuss regularized quantile regression with the elastic net penalty and the group lasso penalty. Both simulated and real data examples show that Bayesian regularized quantile regression methods often outperform quantile regression without regularization and their non-Bayesian counterparts with regularization