Exploiting Similarity of Structure in Hierarchical Reinforcement Learning with QBOND

As robots become increasingly common in modern society, the need for effective machine learning of robot tasks is becoming more significant. Hierarchical Reinforcement Learning (HRL) has been shown to improve the tractability of learning on specific problem sets, but does not yet effectively solve many open, yet repetitive problem environments. This thesis extends HRL theory to be less reliant on specific environmental landmarks, and expands the set of problems that HRL can efficiently solve. This thesis introduces a new HRL approach called QBOND. QBOND allows the value function of a Markov Decision Process to be safely decomposed despite wide, well-connected boundaries between subtasks. It also exploits repetition in the underlying transition and reward functions to reuse subtasks across open, repetitive problems. QBOND's lack of dependence on landmarks for decomposition allows a problem to be decomposed along any arbitrary boundary in the state space. This creates a new set of opportunities for the designer, who can arbitrarily choose where one subtask should end and the next subtask should begin. This thesis presents two different methods for placing boundaries between subtasks. The first minimises the total learning required across the entire hierarchy, helping address the Curse of Dimensionality. The second minimises the total number of physical interactions required, helping address the challenge of acquiring sufficient real world samples for effective learning. The generic nature of QBOND subtasks makes learning an effective set of policies for each subtask different to most HRL approaches. Each subtask does not have a single subgoal at its boundary, so the designer is able to learn policies that are useful in a variety of different situations. This thesis explores four different methods for learning a robust, generic set of policies. These policies can be reused in multiple different contexts within a single problem or even across multiple related problems. A full procedure for applying QBOND to a problem is presented, along with results for 3 different simulated robot control problems