Ergodic and Adaptive Control of Hidden Markov Models∗

A partially observed stochastic system is described by a discrete time pair of Markov processes. The observed state process has a tran-sition probability that is controlled and depends on a hidden Markov process that also can be controlled. The hidden Markov process is com-pletely observed in a closed set and only observed through the other process in the complement of the closed set. Initially an ergodic control problem is formulated and solved by a vanishing discount approach. An equation is given for the optimal cost and an optimal control for the ergodic control problem that is obtained from the family of Bell-man equations for the discounted problems. For the ergodic control problem, the closed set where the hidden Markov process is completely observed can be the empty set. For an adaptive control problem, it is assumed that the transition operators for the observed state process and the hidden Markov process depend on a parameter from a compact metric space. It is shown that the optimal cost is a continuous function of the parameter and there is a control that is almost optimal for a parameter in an open set. It is assumed that the unknown parameter is a random variable. A sequence of estimates of this parameter are given which is modified at random times by the use of a control from a finite family. It i