Scaling Up Robust MDPs using Function Approximation

We consider large-scale Markov decision pro-cesses (MDPs) with parameter uncertainty, un-der the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handling uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the “curse of di-mensionality”, MDPs that model real-life prob-lems are typically prohibitively large for such ap-proaches. In this work we employ a reinforce-ment learning approach to tackle this planning problem: we develop a robust approximate dy-namic programming method based on a projected fixed point equation to approximately solve large scale robust MDPs. We show that the proposed method provably succeeds under certain techni-cal conditions, and demonstrate its effectiveness through simulation of an option pricing problem. To the best of our knowledge, this is the first at-tempt to scale up the robust MDP paradigm. 1