Uncertainty Propagation for Efficient Exploration in Reinforcement Learning

Abstract. Reinforcement learning aims to derive an optimal pol-icy for an often initially unknown environment. In the case of an unknown environment, exploration is used to acquire knowledge about it. In that context the well-known exploration-exploitation dilemma arises—when should one stop to explore and instead ex-ploit the knowledge already gathered? In this paper we propose an uncertainty-based exploration method. We use uncertainty propaga-tion to obtain the Q-function’s uncertainty and then use the uncer-tainty in combination with the Q-values to guide the exploration to promising states that so far have been insufficiently explored. The uncertainty’s weight during action selection can be influenced by a parameter. We evaluate one variant of the algorithm using full covari-ance matrices and two variants using an approximation and demon-strate their functionality on two benchmark problems.