DRAFT for chapter on NN Adaptive Critics in Karayiannis book 1/12/99 11:14pm> Page 1 DESIGNING (APPROXIMATE) OPTIMAL CONTROLLERS via DHP ADAPTIVE CRITICS & NEURAL NETWORKS

The objective of this chapter is to provide the reader some guidance in applying the Dual Heu-ristic Programming (DHP) method in the context of designing neural-network controllers. DHP is a member of the class of Critic methods, which in turn is a member of the class of Reinforcement Learning methods. Development of the DHP method benefited from the confluence of several other developments; the following subsections describe associated background ideas useful in appreciating the DHP method. Subsequent sections will describe the DHP method itself, provide suggestions for application of DHP, and present worked-out examples. 1.1 Learning Algorithms A key distinguishing feature of the computational paradigm known as neural networks is its attribute of attaining knowledge via interaction with its environment (vs. having knowledge pro-grammed in). A significant area of research in the neural network (NN) field has been, and con-tinues to be, that of developing strategies by which the NN accomplishes this extraction of knowledge from its environment-- these are typically referred to as learning algorithms. These algorithms fall into three general categories (in the following descriptions, the terms ‘pupil ’ and ‘teacher ’ designate, respectively, the NN that is learning, and the process used to accomplish the learning; further, except where indicated, the pupil NN is considered an input/output devise): 1. Supervised Learning: This category entails the teacher role having at its disposal full knowledge of the problem context (about which the pupil NN is to learn), and in particular, has available a collection of data pairs (comprising input and associated desired output) with which to conduct the pupil’s learning process