A COMPARATIVE STUDY OF DISCRETIZATION APPROACHES 1 Chapter 1 A COMPARATIVE STUDY OF DISCRETIZATION APPROACHES FOR STATE SPACE GENERALIZATION IN THE KEEPAWAY SOCCER TASK

There are two main branches of reinforcement learning: methods that search di-rectly in the space of value functions that asses the utility of the behaviors (Temporal Difference Methods); and methods that search directly in the space of behaviors (Pol-icy Search Methods). When applying Temporal Difference (TD) methods in domains with very large or continuous state spaces, the experience obtained by the learning agent in the interaction with the environment must be generalized. The generalization can be carried out in two different ways. On the one hand by discretizing the environ-ment to use a tabular representation of the value functions (e.g. Vector Quantization Q-Learning algorithm). On the other hand, by using an approximation of the value functions based on a supervised learning method (e.g. CMAC Q-Learning algorithm). Other algorithms use both approaches to benefit from both mechanisms, allowing a higher performance. This is the case of the Two Step Reinforcement Learning algo-rithm. In the case of Policy Search Methods, the Evolutionary Reinforcement Learn-ing algorithm has shown promising in RL tasks. All these algorithms present different ways to tackle the problem of large or continuous state spaces. In this chapter, we or-ganize and discuss different generalization techniques to solve this problem. Finally, we demonstrate the usefulness of the different algorithms described to improve the learning process in the Keepaway domain. A COMPARATIVE STUDY OF DISCRETIZATION APPROACHES 3 1