Learning 2D Linear Dynamics in Image Space Using Neural Networks

∗ denotes equal contribution Abstract—Grasping and manipulating a previously unknown object without assumptions about structure or relevant features in the environment is an especially challenging problem in robotics. Motivated by recent work on feature learning with deep neural networks, we approach this problem by learning a dynamical system that relates robotic control inputs (e.g., motor torques) to features learned from raw sensory data (e.g., images) using a deep neural network. We present an algorithm for inferring the most likely parameters of this system using approximate expectation maximization. Initial experiments show that the method is able to predict the next image of a pendulum in a 2D simulator to less than 1 pixel of mean squared error using only the current image and the control inputs to the system. I