Facial gesture analysis in an interactive environment

UnrestrictedThis research focuses on tracking, modeling, quantifying and analyzing facial motions for gesture understanding. Facial gesture analysis is an important problem in computer vision since facial gestures carry signals besides words and are critical for nonverbal communication. The difficulty of automatic facial gesture recognition lies in the complexity of face motions. These motions can be categorized into two classes: global, rigid head motion, and local, nonrigid facial deformations. In reality, observed facial motions are a mixture of these two components.; In this work, we propose a framework to take both of these two motions into account. The whole framework consists of three components: 3D head pose estimation, modeling local deformations, and expression classification.; We propose a novel hybrid 3D head tracking algorithm to differentiate these two motions. The hybrid tracker integrates both intensity and feature correspondence information for robust real-time head pose estimation. Based on this tracker, we classify video segments into expressions by learning a graphical representation for nonrigid facial motions. The graphical model characterizes each face region by dense motion fields, and encodes inter-region dependencies in joint density functions. This graphical model is learned empirically using the EM algorithm and expression recognition is performed by Bayesian MAP estimation.; In addition, rigid and nonrigid motions are analyzed simultaneously in 3D by manifold learning techniques. We decompose nonrigid facial deformations on a basis of 1D manifolds. Each 1D manifold is learned offline from sequences of labeled basic expressions, such as smile, surprise, etc. Any expression is then a linear combination of values along these axes, with coefficient representing the level of activation. We experimentally verify that expressions can indeed be represented this way, and that individual manifolds are indeed 1D. The manifold learning and dimensionality estimation are all implemented in the N-D Tensor Voting framework. The output of our system is a rich representation of the face, including the 3D pose, 3D shape, expression label with probability, and the activation level