Variational policy search via trajectory optimization

In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task. While random exploration can work well in simple domains, complex and high-dimensional tasks present a serious challenge, particularly when combined with high-dimensional policies that make parameter-space exploration infeasible. We present a method that uses trajectory optimization as a powerful exploration strat-egy that guides the policy search. A variational decomposition of a maximum likelihood policy objective allows us to use standard trajectory optimization al-gorithms such as differential dynamic programming, interleaved with standard supervised learning for the policy itself. We demonstrate that the resulting algo-rithm can outperform prior methods on two challenging locomotion tasks.