Causal Action: A Framework to Connect Action Perception and Understanding

Human actions are more than mere body movements. In contrast to dynamic events involving inanimate objects, human actions have a special status in that they control interactions with the world and afford privileged access to the experience of agency and to control interactions with the world. Several causal constraints on human actions support the generation and the understanding of actions. For example, human actions inherently involve a causal structure: limb movements generally cause changes in body position along a path through the environment to achieve intentional goals. However, it remains unclear how the system that supports action perception communicates with high-level reasoning system to recognize actions, and more importantly, to achieve a deeper understanding of observed actions. My dissertation aims to determine whether causality imposes critical motion constraints on action perception and understanding, and how causal relations involved in actions impact behavioral judgments. The project also investigates the developmental trajectory and neural substrate of action processing, and whether a feedforward deep learning model is able to learn causal relations solely from visual observations of human actions. Through behavioral experiments, an infant eye movement study, a neural study using magnetoencephalography, and model simulations, my dissertation yields a number of insights. 1) Humans implicitly and automatically rely on causal expectations to explain motion information when perceiving body movements and meaningful social interactions; 2) Sensitivity to causal constraints on actions develops early in infants even before 18 months of age, and shows a clear relation with the development of gross motor functions. 3) Congruency to causal relations involved in human actions can be decoded from neural MEG signals, with the processing of causal actions eliciting a distributed neural network. The brain network involves temporal, parietal, and frontal regions that are important locus for spatial relation reasoning, decision making, and intention understanding. 4) Recent stimulus-driven deep learning neural networks are unable to learn the causal relations involved in actions, failing to create high-level representations for causal actions. Overall, my dissertation reveals the importance of causality in bridging action perception and understanding, highlights the key missing computational components in deep learning models for complex visual stimuli such as human actions, and provides a potential framework to connect seeing and thinking