Reinforcement learning using reservoir computing for soft robotic control: A bio-inspired system

Modern approaches in machine learning and artificial intelligence are dominated by deep learning. Although inspired by the brain, these network architectures are not biologically plausible. In contrast, the reservoir computing paradigm has a single sparsely and recurrently connected hidden layer, with the linear readout layer being the only learned parameter. We apply reservoir computing in a reinforcement learning context to actuate a soft, slender muscular arm. The arm must track a moving target in a partially observable environment. Soft robots present an especially challenging test bed for reinforcement learning due to their nonlinear, continuum dynamics. We propose learning strategies for two classes of reservoirs: Echo State Networks (ESNs) built using tanh neurons and Liquid State Machines (LSMs) built using Leaky Integrate and Fire (LIF) neurons. Unlike traditional activation functions, LIF neurons provide discrete spike trains with respect to time, and LSM-like structures are found in vivo. Crucially, we prohibit any additional feedforward layers, making the reservoir the sole neural computing unit. Our ESN policy significantly outperforms the state-of-the-art algorithm Proximal Policy Optimization (PPO) and our LSM policy matches PPO. Finally, we deploy a LSM directly on neuromorphic hardware, opening up opportunities for energy efficient reinforcement learning and robotic control. End-to-end, our soft robot controlled with a spiking reservoir is a novel bio-inspired system.LimitedAuthor requested closed access (OA after 2yrs) in Vireo ETD syste