Sensorimotor Aspects of Brain Function: Development, Internal Dynamics, and Tool Use

Learning through the sensorimotor loop is essential for intelligent agents. While the important role of sensorimotor learning has been studied, several important aspects of sensorimotor learning in the brain, such as the development of motor behavior maps, the influence of internal dynamics on external behaviors, and the emergence of tool-use capabilities, are not addressed significantly. In this dissertation, I will address three questions trying to probe the nature of sensorimotor learning: (1) How can a sensorimotor agent understand its own body by developing a cortical map of its motor actions?; (2) How can predictive internal brain dynamics play an important role in external behavior (authorship of actions)?; (3) How tool-use emerges in a sensorimotor system interacting with the environment? The first topic considers developing a cortical motor action map in a sensorimotor agent. Motivated by an experimental study showing a topographical map of complex behaviors in the macaque brain, we developed a target reaching gesture map using a biologically motivated self-organizing map model of the cortex with two-joint arm movements as inputs. The resulting gesture map showed a global topographic order based on the target locations. The map is comparable to the motor map reported in the experimental study. In the second topic of this dissertation, I discuss and investigate the role of the predictive internal brain dynamics. Previous computer simulation studies showed that neural network controllers with more predictable internal state dynamics can attain higher performance in harsher or changing environments. This implies that predictable internal state dynamics could be a necessary condition for intelligent agents in the evolutionary pathway to have authorship of actions and to adapt themselves to changing environment. However, there was a missing link; the findings from the simulation study do not necessarily mean that the internal state dynamics affects the external behavior (authorship of actions) in the biological brain as well. To fill the gap, I investigated the role of predictability of internal state dynamics in the brain by analyzing the human EEG data. These results support our hypothesis on the existence of predictable dynamics and its relation to conscious states (as a surrogate of authorship of actions). Lastly, in the third topic of this dissertation, I present tool-use in sensorimotor agents. Tool-use requires high levels of sensorimotor skill learning and problem solving capabilities and is one of the salient indicators of intelligence along with communication (language) and logic. However, through our literature search, we found that there are two gaps in tool-use in AI and robotics. First, most works depend on some degree of designer knowledge regarding tool-use and motor control. Furthermore, tool-use tasks that require multiple subtasks to be completed in specific order are almost non-existent in deep reinforcement learning (RL) literature for developments and benchmarks, even though deep RL has demonstrated good performance in some control tasks in recent years. In this dissertation, I present environments and sensorimotor systems where the agents can adapt to use simple or complex tools based on minimal task knowledge. Specifically, I present two approaches. I first evolve neural network controllers for simple tool-use behavior in reaching tasks with minimal task knowledge, followed by analysis of the evolved networks. The results show that minimal, indirect fitness criteria are enough to give rise to tool-use behavior. Then, as a second step, I implement a more complex tool-use environment such as dragging an object to a target location using a tool in a physics simulation, and demonstrate a deep RL method with stepwise composite reward shaping methodology to learn the complex tool-use task successfully. Overall, the studies in this dissertation are expected to help researchers in brain science and AI understand the nature of sensorimotor aspects of learning in the brain and implement them in AI