Imitation and Reinforcement Learning in Agents with Heterogeneous Actions

Abstract. Reinforcement learning techniques are increasingly being used to solve dicult problems in control and combinatorial optimization with promising results. Implicit imitation can accelerate reinforcement learn-ing (RL) by augmenting the Bellman equations with information from the observation of expert agents (mentors). We propose two extensions that permit imitation of agents with heterogeneous actions: feasibility testing, which detects infeasible mentor actions, and k-step repair, which searches for plans that approximate infeasible actions. We demonstrate empirically that both of these extensions allow imitation agents to con-verge more quickly in the presence of heterogeneous actions.