Facial Emotion Recognition in Continuous Video

Facial emotion recognition–the detection of emo-tion states from video of facial expressions–has appli-cations in video games, medicine, and affective com-puting. While there have been many advances, an ap-proach has yet to be revealed that performs well on the non-trivial Audio/Visual Emotion Challenge 2011 data set. A majority of approaches still employ single frame classification, or temporally aggregate features. We as-sert that in unconstrained emotion video, a better clas-sification strategy should model the change in features, versus simply combining them. We compute a deriva-tive of features with histogram differencing and deriva-tive of Gaussians and model the changes with a hidden Markov model. We are the first to incorporate temporal information in terms of derivatives. The efficacy of the approach is tested on the non-trivial AVEC2011 data set and increases classification rates on the data by as much as 13%. 1