Towards convergent offline reinforcement learning

Reinforcement learning (RL) is a leading method for automated sequential decision-making. However, RL is rarely used in the real world. This neglect is due to instability and inefficiency, often even occurring on benchmark problems. Conversely, the sweeping real-world impact of supervised learning (SL) began years ago. Given how much more powerful RL can be than SL, the value of making RL stable and efficient would be immense. Since we know SL is stable and efficient, one candidate for making RL more stable and efficient is to approach RL more like we approach SL. Indeed, RL algorithms with SL-like convergence guarantees already help stability, and RL algorithms designed to learn from offline datasets (static datasets) already help sample efficiency. In this thesis, we find that combining the two performs far better than expected, and we attempt to figure out why. We conclude with ideas for fixing one of the most straightforward convergent RL algorithms.Computer Science