Semantics-guided Human Motion Modeling in Virtual Reality Environment

Human Motion Modeling is essential in Computer Animation and Human-Computer Interaction. This dissertation studies how to enhance the speed and robustness of Human Motion Modeling in Virtual Reality (VR) environments. Specifically, we aim to design a pipeline to effectively capture and use semantic action information to guide the motion capturing from users in physical worlds and its transfer onto digital avatars in VR environments. To recognize the user\u27s action, we first proposed a new Dynamic Directed Graph Convolutional Network (DDGCN) to model spatial and temporal features from users\u27 skeletal representations. The DDGCN consists of several dynamic feature modeling modules to capture effective spatial-temporal action features. To utilize the action semantics to model ongoing actions in real-time, actions need to be recognized as early as possible, before they are finished. Therefore, we studied early action detection and proposed a new pipeline with two new technical designs to make feature learning more robust against temporal scale variance in action sequences and against ambiguity in similar-looking actions. With recognized/predicted action semantics, we proposed a semantics-guided motion retargeting algorithm that uses semantic action information to constrain the motion retargeting. Experimental results demonstrated promising results and effectiveness of our entire proposed pipeline in real-time human motion modeling and reconstruction