Reinforcement Learning for Agile Locomotion: From Algorithms to Design Tools

Reinforcement learning (RL) provide a potentially powerful framework for designing control strategies that enable robots and simulated digital creatures to learn to move with skill and grace. However, there are significant drawbacks from a design perspective: reward functions can be unintuitive, solutions are prone to local minima and hyperparameter choices, there is no direct support for iterative design, and the transfer of motions from simulation to the real world is uncertain. We present a number of insights and refinements in support of learning realistic, controllable movements. These include motion mimicry, multi-step iterative design, sample-based transfer learning, and hybrid learning that mixes supervised learning with policy gradients. We demonstrate simulated human and animal skills that can reproduce a large variety of highly dynamic motions. We further show successful sim2real transfer of dynamic locomotion to Cassie, a large bipedal robot produced by Agility Robotics. Lastly, we highlight recent work by others that builds on key aspects of these ideas, including learning skills from video and the control of full-body muscle-driven motions.Non UBCUnreviewedAuthor affiliation: University of British ColumbiaFacult