Markov decision processes and quasi-martingales

It is showed in this paper that quasi-(super)martingales play an important role in the theory of Markov decision processes. For excessive functions (with respect to a charge) it is proved that the value of the state at time t converges almost surely under each Markov strategy, which implies that the value function in the state at time t converges to zero (a.s), if an optimal strategy is used. At last a characterization of the conserving and equalizing properties is formulated using martingale theory