Hierarchical Deep Reinforcement Learning For Robotics and Data Science

This dissertation explores learning important structural features of a Markov DecisionProcess from offline data to significantly improve the sample-efficiency, stability, and robustnessof solutions even with high dimensional action spaces and long time horizons. Itpresents applications to surgical robot control, data cleaning, and generating efficient executionplans for relational queries. The dissertation contributes: (1) Sequential WindowedReinforcement Learning: a framework that approximates a long-horizon MDP with a sequenceof shorter term MDPs with smooth quadratic cost functions from a small numberof expert demonstrations, (2) Deep Discovery of Options: an algorithm that discovers hierarchicalstructure in the action space from observed demonstrations, (3) AlphaClean: asystem that decomposes a data cleaning task into a set of independent search problemsand uses deep q-learning to share structure across the problems, and (4) Learning QueryOptimizer: a system that observes executions of a dynamic program for SQL query optimizationand learns a model to predict cost-to-go values to greatly speed up future searchproblems