BAYESIAN PARAMETER ESTIMATION AND ADAPTIVE CONTROL OF MARKOV PROCESSES

Abstract. This paper considers Bayesian parameter estimation and an associated adaptive control scheme for controlled Markov chains and diffu-sions with time-averaged cost. Asymptotic behaviour of the posterior law of the parameter given the observed trajectory is analyzed. This analysis sug-gests a “cost-biased ” estimation scheme and associated self-tuning adaptive control. This is shown to be asymptotically optimal in the almost sure sense. I. Introduction. A popular scheme for adaptive control is the so-called “self-tuning ” control wherein a parameterized family of system models is presupposed and the parameter is estimated “on-line”. One then uses at each time instant that control which would have been the optimal choice for the current value of the system state if the current parameter estimat