Improved computational methods for Bayesian tree models

Trees have long been used as a flexible way to build regression and classification models for complex problems. They can accommodate nonlinear response-predictor relationships and even interactive intra-predictor relationships. Tree based models handle data sets with predictors of mixed types, both ordered and categorical, in a natural way. The tree based regression model can also be used as the base model to build additive models, among which the most prominent models are gradient boosting trees and random forests. Classical training algorithms for tree based models are deterministic greedy algorithms. These algorithms are fast to train, but they usually are not guaranteed to find an optimal tree. In this paper, we discuss a Bayesian approach to building tree based models. In Bayesian tree models, each tree is assigned a prior probability based on its structure, and standard Monte Carlo Markov Chain (MCMC) algorithms can be used to search through the posterior distribution. This thesis is aimed at improving the computational efficiency and performance of Bayesian tree based models. We consider introducing new proposal or “moves” in the MCMC algorithm to improve the efficiency of the algorithm. We use temperature based algorithms to help the MCMC algorithm get out of local optima and move towards the global optimum in the posterior distribution. Moreover, we develop semi-parametric Bayesian additive trees models where some predictors enter the model parametrically. The technical details about using parallel computing to shorten the computing time are also discussed in this thesis