Analysis of facial and body feature point trajectories for automatic affect recognition

Affective computing systems must perform affect recognition both automatically and under few constraints to be useful for real-world applications. This thesis addresses the research problem of effectively performing automatic affect recognition using facial and body expressions in images and video under few constraints. Expression recognition is challenging due to the difficulty of extracting affective features that effectively represent visual cues in the presence of much variation, e.g., subject pose view, subject size and subject-to-camera distance. Novel affective algorithms are proposed that: 1) perform expression normalisation to produce affective features that facilitate expression recognition under few constraints; 2) form emotion subcategories (based on positions of normalised feature points) to group similar expressions in basic emotion categories, facilitating intra-class classification of expressions; and 3) analyse normalised feature point movement to identify affective segments of expression sequences and utilise expression dynamics to improve recognition performance. The proposed automatic static expression recognition systems utilise novel geometric features based on the trajectory of feature points between peak expressions and neutral expression models. Normalisation algorithms significantly reduce the impact of many variations in subject movement and other factors. Emotion subcategories are formed by automatically grouping similar expressions within each basic emotion category. Classification is performed using two-stage classifiers that utilise emotion subcategories to analyse variations in expression within each basic emotion. Dynamic extensions of the expression recognition systems are proposed that analyse feature point movement to automatically identify affective segments of expression sequences and to form dynamic features for affect recognition. The proposed emotion recognition systems outperform similar systems in published work, showing the effectiveness of the proposed expression feature and classification processes. Analysis of feature point movement allows affective segments of expression sequences to be both reliably and accurately located. The proposed dynamic affect recognition systems utilise temporal affective information to achieve improvements in recognition performance over their static counterparts