Learning Discriminative Feature Representations for Action Recognition and Segmentation

The recognition of actions and activities has a long history in the computer vision community. State-of-the-art approaches are based on analysis of the distribution of low-level feature descriptors such as space-time interest points and dense trajectories. These feature descriptors are typically handcrafted local features, thus they are not interpretable and discriminative enough. Recent advances in deep learning have led to improved performance in action recognition. However, most deep learning techniques are directly borrowed from the image classification literature and do not take into consideration spatial constraints on the structure of the human body or temporal constraints on action transitions. Furthermore, while most of the existing methods assume that videos/sequences are trimmed so that each sequence contains only one instance of the action, in real-world scenarios, a sequence may contain multiple actions that change in time. Therefore joint recognition and segmentation is required. In this thesis dissertation, we explore data-driven models for the task of action recognition and action segmentation. We first propose an interpretable human body-part based model for action recognition using a two-layer feature-learning framework. This model can automatically discover discriminative body-part movement patterns for each action, and learn these patterns jointly with action classifiers. For the task of joint action recognition and segmentation, we extend the standard Conditional Random Field (CRF) model to incorporate top-down information about each action segment using a hybrid Markov/semi-Markov model and show improved performance when compared with chain CRF models. Furthermore, to learn discriminative features for temporal segmentation, we incorporate an additional feature representation layer using two different frameworks. The first one, called Sparse Hidden Markov Model, is a generative model that integrates Hidden Markov Model with sparse dictionary learning. The second one, called Hierarchical CRF, is a discriminative model that integrates CRFs with a neural network. We show that both methods achieve superior performance by learning the feature representation jointly with temporal models. Moreover, the proposed hierarchical CRF model achieves the best performance relative to the state-of-the-art methods