Learning Hierarchical Policies from Human Feedback

Robots are on the verge of becoming ubiquitous. In the form of affordable humanoid toy robots, autonomous cars, vacuum robots or quadrocopters, robots are becoming part of our everyday life. As of today, most of these robots still follow largely hard coded behavior routines. Constraining a robot’s behavior to pre-programmed routines, however, limits its potential in several important ways. For example, programming even simple behavior patterns is a challenging task and programming behavior with human like performance by hand seems impossible. The goal of this thesis, thus, is to develop methods which allow robots to learn solutions to tasks through trial and error instead of relying on manual programming. These learned solutions should fulfill a range of desired properties. Foremost, the solutions should be learned on real world robots and not be constrained to simplified simulation environments. Furthermore, we would like the robot to learn versatile solutions which are able to cope with different variations of a task. Finally, we would like to be able to also solve more complicated tasks which require sequencing of multiple skills. In the first part of this thesis, we propose an algorithm to learn such versatile solutions. The proposed algorithm aims to find a hierarchical policy, consisting of a gating policy and a set of sub-policies. The gating policy selects a sub-policy and the sub-policy decides which action to take. Each of the learned sub-policies may be able to solve the task or a part of the overall task. To learn multiple sub-policies from one sample set, we employ an expectation-maximization based learning algorithm, where the sub-policies are updated according to their responsibilities for individual samples. These responsibilities indicate how likely it is that a certain sub-policy generated a given state-action pair. By constraining our learning algorithm to solutions which increase the entropy of the responsibilities, the robot will learn a set of sub-policies that encode different solutions for the same task. This kind of concurrency is highly desirable as it allows the robot to learn back-up solutions which may still be valid even if the original solution fails due to changes in the robot or environment. In the second part of this thesis, we tackle the challenge of learning skills directly from state- action trajectories. A common approach in robot learning is to assume access to some sort of parametrized skill to represent the robot’s behavior over multiple time steps. These skills are usually either movement primitives (MPs) or parametrized feedback controllers. While especially MPs have been a cornerstone in advancing the state of the art in robot learning, it is not clear how to learn from the actions taken throughout the execution of a skill and MPs usually do not encode feedback. In the discrete state-action reinforcement learning (RL) setting, macro-actions, or options, have been introduced to address the problem of learning temporally correlated actions, which can be viewed as a form of skill. To connect robot learning with the advances in the field of discrete state-action RL, we propose a probabilistic framework to infer options from state- action trajectory observations. The inference is based on a hidden Markov model (HMM), where the options indices are modeled as latent variables and where inference can be performed by adapting well known expectation-maximization algorithms. Because this framework allows for the inference of parametric policies, it is also compatible with policy search (PS) methods, a class of RL algorithms which is at the core of many recent successes in robot learning. Learning methods, such as the one we propose in the first chapter of this thesis, enable robots to learn from trial and error instead of having to program specific solutions. Unfortunately, the quality of the learned solutions still depends heavily on a significant amount of expert knowledge and programming which has to be directed at designing reward functions. In RL, reward functions effectively serve as task description, guiding the robot to a good solution by encoding the desir- ability of states and actions. Designing these reward functions, however, is a difficult task even for experts and it is unlikely that non-experts will be able to train robots on new tasks by means of programming reward functions. Thus, in the last part of this thesis, we propose a method which allows the robot to model a teacher’s implicit reward function during the RL process. This reward model is represented by a Gaussian process (GP), a class of probabilistic function approximators which allow us to take both the teacher’s and the robot’s uncertainty into account. Using the proposed method, the robot can ask the teacher to evaluate the quality of just a few select robot actions, such that the robot can improve its reward model. To minimize the amount of human- robot interactions, the robot uses its uncertainty estimate to request evaluations only for actions which seem to be promising but have a high uncertainty. In this setting, the teacher will not need to be a roboticist and no manual coding is required. Altogether, the individual contributions of this thesis allow robots to learn versatile solutions to complicated tasks from trial and error. These solutions generalize over similar settings and are robust by representing multiple solutions to one task. The robot is able to learn these solutions by interacting with a human teacher, eliminating the need for task specific expert programming. The contributions of this thesis are evaluated on a wide array of both simulated and real world tasks and the results show the effectiveness of the proposed methods