Ergodic Theory for Controlled Markov Chains with Stationary Inputs

International audienceConsider a stochastic process X on a finite state space X = {1,. .. , d}. It is conditionally Markov, given a real-valued 'input process' ζ. This is assumed to be small, which is modeled through the scaling, ζ t = εζ 1 t , 0 ≤ ε ≤ 1 , where ζ 1 is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on ζ: (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain X • obtained with ζ ≡ 0. The triple (X, X • , ζ) is a jointly stationary process satisfying P{X(t) = X • (t)} = O(ε) Moreover, a second-order Taylor-series approximation is obtained: P{X(t) = i} = P{X • (t) = i} + ε 2 π (2) (i) + o(ε 2), 1 ≤ i ≤ d, with an explicit formula for the vector π (2) ∈ R d. (ii) For any m ≥ 1 and any function f : {1,. .. , d} × R → R m , the stationary stochastic process Y (t) = f (X(t), ζ(t)) has a power spectral density S f that admits a second order Taylor series expansion: A function S (2) f : [−π, π] → C m×m is constructed such that S f (θ) = S • f (θ) + ε 2 S (2) f (θ) + o(ε 2), θ ∈ [−π, π] in which the first term is the power spectral density obtained with ε = 0. An explicit formula for the function S (2) f is obtained, based in part on the bounds in (i). The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu