Exploration of the Practical Issues of Learning Prediction-Control Tasks Using Temporal Difference Learning Methods

in Computer Science There has been recent interest in using a class of incremental learning algorithms called temporal difference learning methods to attack problems of prediction. These algorithms have been brought to bear on various prediction problems in the past, but have remained poorly understood. It is the purpose of this thesis to further explore this class of algorithms, particularly the TD (! ) algorithm. A number of practical issues are raised and discussed from a general theoretical perspective and then explored in the context of several case studies. The thesis presents a framework for viewing these algorithms independent of the particular task at hand and uses this framework to explore not only tasks of prediction, but also prediction tasks that require control, whether complete or partial. This includes applying the TD (!) algorithm to two tasks: 1) learning to play tic-tac-toe from the outcome of self-play and the outcome of play against a perfectly-playing opponent and 2) learning two simple one-dimensional image segmentation tasks