Learning a Sparse Codebook of Facial and Body Microexpressions for Emotion Recognition

Obtaining a compact and discriminative representation of facial and body expressions is a difficult problem in emotion recognition. Part of the difficulty is capturing microexpressions, i.e., short, invol-untary expressions that last for only a fraction of a second: at a micro-temporal scale, there are so many other subtle face and body movements that do not convey semantically meaningful informa-tion. We present a novel approach to this problem by exploiting the sparsity of the frequent micro-temporal motion patterns. Local space-time features are extracted over the face and body region for a very short time period, e.g., few milliseconds. A codebook of mi-croexpressions is learned from the data and used to encode the fea-tures in a sparse manner. This allows us to obtain a representation that captures the most salient motion patterns of the face and body at a micro-temporal scale. Experiments performed on the AVEC 2012 dataset show our approach achieving the best published per-formance on the expectation dimension based solely on visual fea-tures. We also report experimental results on audio-visual emotion recognition, comparing early and late data fusion techniques