Learning Multidimensional Control Actions From Delayed Reinforcements

This paper addresses the problem of learning multidimensional control actions from delayed rewards. Classical reinforcement learning algorithms can be applied to tasks with multidimensional action spaces by recoding the action space appropriately (transforming it artificially to a single dimension), but this straightforward recoding approach suffers from significant inefficiencies. An alternative approach to applying Q-learning to tasks with vector actions is proposed, called Q-V-learning. Experimental results are presented where this algorithm clearly outperforms the simple recoding approach, while it is associated with a much lower computational expense. INTRODUCTION The basic scenario of the reinforcement learning (RL) [8, 11, 4, 1] paradigm is as follows. At each time step the learning agent observes the current state of its environment and performs an action. Then it receives a reinforcement value, also called a payoff or a reward, and a state transition takes place. State transi..