Human Action Recognition based on Probabilistic Graphical Models

Video-based human action recognition is a key component of video analysis. Despite significant research efforts over the past few decades, human action recognition still remains a challenging problem. My dissertation investigates three important and emerging topics in practical action recognition problems. First, we consider a crucial problem of recognizing actions having similar movements. An approach called Action Trait Code (ATC) for human action classification is proposed to represent an action with a set of velocity types derived by the averages velocity of each body part. An effective graph model based on ATC is employed for learning and recognizing human actions. To examine recognition accuracy, we evaluate our approach on our self-collected action database. Second, an action video may have many similar observations with occasional and irregular changes, which make commonly used fine-grained features less reliable. This dissertation introduces a set of temporal pyramid features that enriches action representation with various levels of semantic granularities. For learning and inferring the proposed pyramid features, we adopt a discriminative model with latent variables to capture the hidden dynamics in each layer of the pyramid. Experimental results show that our method achieves more favorable performance than existing methods. Third, actions taken in real-world applications often come with corrupt segments of frames caused by various factors. These segments may be of arbitrary lengths and appear irregularly. They change the appearance of actions dramatically, and hence degrade the performance of a pre-trained recognition system. In this dissertation, I presented an integrated approach which includes two key components {em outlier filtering} and {em observation completion}. It can tackle this problem without making any assumptions about the locations of the unobserved segments. Specifically, the outlier frame filtering mechanism is introduced to identify the unobserved frames. Our observation completion algorithm is designed to infer the unobserved parts. It treats the observed parts as the query to the training set, and retrieves coherent alternatives to replace the unobserved parts. Hidden conditional random fields (HCRFs) are then used to recognize the filtered and completed actions. Furthermore, we collect a new action dataset where outlier frames irregularly and naturally present. Besides this dataset, our approach is evaluated on two benchmark datasets, and compared with several state-of-the-art approaches. The superior results obtained by using our approach demonstrate its effectiveness and general applicability. On the other hand, I also develop a unified action recognition framework that can jointly handle the outlier detection and predicting actions. For each action to be recognized, we explore the mutual dependency between its frames, and augment each frame with extra alternative frames borrowed from training data. The augmentation mechanism is designed in the way where a few alternatives are of high quality, and can replace the detected corrupt frames. Our approach is developed upon hidden conditional random fields. It integrates corrupt frame detection and alternative selection into the process of prediction, and can more accurately recognize partially observed actions. The promising results manifest its effectiveness and large applicability