The robustness-performance tradeoff in Markov decision processes

Computation of a satisfactory control policy for a Markov decision process when the parameters of the model are not exactly known is a problem encountered in many practical applications. The traditional robust approach is based on a worst-case analysis and may lead to an overly conservative policy. In this paper we con-sider the tradeoff between nominal performance and the worst case performance over all possible models. Based on parametric linear programming, we propose a method that computes the whole set of Pareto efficient policies in the performance-robustness plane when only the reward parameters are subject to uncertainty. In the more general case when the transition probabilities are also subject to error, we show that the strategy with the “optimal ” tradeoff might be non-Markovian and hence is in general not tractable.