Bayesian methods for knowledge transfer and policy search in reinforcement learning

Graduation date: 2013How can an agent generalize its knowledge to new circumstances? To learn\ud effectively an agent acting in a sequential decision problem must make intelligent action selection choices based on its available knowledge. This dissertation focuses on Bayesian methods of representing learned knowledge and develops novel algorithms that exploit the represented\ud knowledge when selecting actions.\ud \ud Our first contribution introduces the multi-task Reinforcement\ud Learning setting in which an agent solves a sequence of tasks. An\ud agent equipped with knowledge of the relationship between tasks can\ud transfer knowledge between them. We propose the transfer of two\ud distinct types of knowledge: knowledge of domain models and knowledge\ud of policies. To represent the transferable knowledge, we propose\ud hierarchical Bayesian priors on domain models and policies\ud respectively. To transfer domain model knowledge, we introduce a new\ud algorithm for model-based Bayesian Reinforcement Learning in the\ud multi-task setting which exploits the learned hierarchical Bayesian\ud model to improve exploration in related tasks. To transfer policy\ud knowledge, we introduce a new policy search algorithm that accepts a\ud policy prior as input and uses the prior to bias policy search. A\ud specific implementation of this algorithm is developed that accepts a\ud hierarchical policy prior. The algorithm learns the hierarchical\ud structure and reuses components of the structure in related tasks.\ud \ud Our second contribution addresses the basic problem of generalizing knowledge gained from previously-executed policies. Bayesian\ud Optimization is a method of exploiting a prior model of an objective function to quickly identify the point maximizing the modeled objective.\ud Successful use of Bayesian Optimization in Reinforcement Learning\ud requires a model relating policies and their performance. Given such a\ud model, Bayesian Optimization can be applied to search for an optimal\ud policy. Early work using Bayesian Optimization in the Reinforcement\ud Learning setting ignored the sequential nature of the underlying\ud decision problem. The work presented in this thesis explicitly\ud addresses this problem. We construct new Bayesian models that take\ud advantage of sequence information to better generalize knowledge\ud across policies. We empirically evaluate the value of this approach in\ud a variety of Reinforcement Learning benchmark problems. Experiments\ud show that our method significantly reduces the amount of exploration\ud required to identify the optimal policy.\ud \ud Our final contribution is a new framework for learning parametric\ud policies from queries presented to an expert. In many domains it is\ud difficult to provide expert demonstrations of desired policies.\ud However, it may still be a simple matter for an expert to identify\ud good and bad performance. To take advantage of this limited expert\ud knowledge, our agent presents experts with pairs of demonstrations and\ud asks which of the demonstrations best represents a latent target\ud behavior. The goal is to use a small number of queries to elicit the\ud latent behavior from the expert. We formulate a Bayesian model of the\ud querying process, an inference procedure that estimates the posterior\ud distribution over the latent policy space, and an active procedure for\ud selecting new queries for presentation to the expert. We show, in\ud multiple domains, that the algorithm successfully learns the target\ud policy and that the active learning strategy generally improves the\ud speed of learning