Learning and shaping in emergent hierarchical control systems

The use of externally imposed hierarchical structures to reduce the complexity of learning control is common. However it is clear that the learning of the hierarchical structure by the machine itself is an important step towards more general and less bounded learning. Presented in this paper is a nested Q-learning technique that generates a hierarchical control structure as the robot interacts with its world. Furthermore, given the frailties of real machines and the long learning times required, it is becoming clear that fully unassisted learning for robots is unrealistic and when one considers the tremendous amount of information that novice humans/animals receive it is also unreasonable. Also, presented in this paper are methods for pre-training and supplying initial guidance to prepare robots for future situations.