Behavior coordination for a mobile robot using modular reinforcement learning

Coordination of multiple behaviors independently ob-tained by a reinforcement learning method is one of the issues in order for the method to be scaled to larg-er and more complex robot learning tasks. Direct com-bination of all the state spaces for individual modules (subtasks) needs enormous learning time, and it causes hidden states. This paper presents a method of modu-lar learning which coordinates multiple behaviors tak-ing account of a trade-off between learning time and performance. First, in order to reduce the learning time the whole state space is classified into two cate-gories based on the action values separately obtained by Q learning: the area where one of the learned behav-iors is directly applicable (no more learning area), and the area where learning is necessary due to the com-petition of multiple behaviors (re-learning area). Sec-ond, hidden states are detected by model fitting to the learned action values based on the information criteri-on. Finally, the initial action values in the re-learning area are adjusted so that they can be consistent with the values in the no more learning area. The method is applied to one to one soccer playing robots. Com-puter simulation and real robot experiments are given to show the validity of the proposed method.