A case for feature-based successor features for transfer in reinforcement learning

Abstract: Successor Features stand at the boundary between modelfree and model-based Reinforcement Learning. By predicting a sum of features instead of a sum of rewards, they enable very efficient transfer learning through the General Policy Improvement Theorem. Recent work has shifted the focus of the feature space from learnt features to a well-chosen set of base rewards. While this framework greatly improves stability, it discards the flexibility to generalize outside the base reward space. In this paper, we aim to rekindle interest in "representation-based" Successor Features for transfer learning, by clarifying the possible design choices and providing simple cases where they prevail. In a robot arm scenario, we find that they more easily transfer to unseen tasks without suffering from instabilities during training. We provide visual interpretation of the learnt features to explain this performance