Self-Segmentation of Sequences

The paper presents an approach for hierarchical reinforcement learning that does not rely on a priori hierarchical structures. Thus the approach deals with a more difficult problem compared with existing work. It involves learning to segment sequences to create hierarchical structures, based on reinforcement received during task execution, with different levels of control communicating with each other through sharing reinforcement estimates obtained by each others. The algorithm segments sequences to reduce non-Markovian temporal dependencies, to facilitate the learning of the overall task. Initial experiments demonstrated the basic promise of the approach. I. Introduction Although hierarchical approaches are evidently important to reinforcement learning, most existing hierarchical RL models either do not involve automatically developing hierarchies (i.e., using pre-determined hierarchies; e.g., Dayan and Hinton 1993, Sutton 1995, Precup et al 1998, Parr and Russell 1997, Dietterich 1..