SAM-D2: Spontaneous affect modeling using dimensionally-labeled data

Human affect is continuous rather than discrete. Various affect dimensions represent emotions better than through the use of categorical labels because they best embody the non-basic and complex nature of everyday human expressions. Moreover, spontaneous data ensures more variety of emotion compared human expressions. Moreover, spontaneous data ensures more variety of emotion compared to acted data as subjects are constrained with their expressions and they are limited to expressing discrete emotions under acted schemes. Thus it is better to engage in the use of spontaneous expressions. The focus of this research is to build multimodal models for spontaneous human affect analysis. This requires a dimensionally-labeled database which is the basis for creating the affect models. Previous studies on spontaneous and dimensionally-labeled data have been undertaken before with induced data. In this study, the use of naturally spontaneous data is explored on using the data of the Filipino Multimodal Emotion Database (FiMED2). FiMED2 is annotated with dimensional labels of valence and arousal values. Inter-coder agreement of continuous data is resolved through statistical methods. Multimodal affect models for the face and voice were built using machine learning algorithms where the Support Vector Machine for Regression performed the best. The results for the voice modality were particularly better in comparison with previous research on continuous data. Decision-level fusion was used to merge the results of the two modalities. Experiments with relation to feature selection and gender difference were also performed