Emotion recognition: from physiological signals to affective states

In recent years, the rapid growth of human computer interaction research has accelerated the improving research interest in emotion recognition field. Emotion recognition in humans is one of the most important research areas of human interaction. This technology is useful in many disciplines, such as neuropsychology, artificial intelligence, human computer interaction, signal processing, affective computing, ethology, military technological R&D, and image processing. There are many crucial steps in emotion recognition using physiological responses. These include data collection, signal preprocessing, feature extraction/selection, classification strategy, and classification result analysis. Among these issues, the feature selection and classification steps are especially important. On the other hand, the importance of data collection and signal preprocessing steps should not be underestimated. Well designed and robust data collection method will help to preserve the data accuracy and consistency, so that the real world events or objects are correctly described by these collected data. Good signal preprocessing methods will further enhance this relationship. This thesis focuses on the development of accurate, robust and generalized emotion recognition systems which take into account all the above mentioned issues. One of the key contributions of this thesis is the development of the Biometric Signature Based (BSB) System. The proposed system uses a novel framework to eliminate the influence from the "individual varieties" occurred at the conventional subject-independent procedures. The additional subject classification process will transform a subject-independent emotion recognition problem into several subject-dependent tasks, which will help to improve the recognition rate by identifying each individual prior to the recognition phase. There are two phases in this method. In the 1st stage, data from each subject is trained into separated statistical subject models. A new incoming sample will be classified to one subject model that best suits its inner structure. In other words, this data sample shows the most similar characteristics to the assigned statistical model. Then in the 2nd stage, a general feature selection plus classification procedure is applied to achieve the task of emotion recognition. The experimental results showed that, the BSB system performs more effective than using conventional methods. In addition, through a mutual test experiment, the system shows a general usage in that it is not required for the users to be trained by the system, as long as there are enough representative subject models stored beforehand. We also extend the concept of BSB system in a more robust way. A new robust version of BSB system has been introduced and derived by using the multivariate t-Distribution model. Compared to the original Guassian mixture model (GMM) version, the new BSB system has advantages in providing better recognition rates and is able to simplify the model complexities of the original one. We found out that it is not required for the system to prepare statistical models for each individual subject, as long as there are enough representative models known (trained) by the system. This leads to another key contribution of this thesis, an improved version of the BSB system, called "The Self-adaptive BSB system (SaBSB)". Instead of directly creating several models based on individual subjects, the self adaptive procedure first assumes the whole data pool as one single statistical model, and then successively splits the model into two new ones, until the model number reaches a pre-defined value. Hence, the total number of statistical models in the first stage is reduced, also the models are generated in a more adaptive and flexible manner. By comparing SaBSB with conventional BSB and robust BSB, the results show that SaBSB achieves relatively comparable results but requires less constraint in the number of subject models to be built. Besides the efforts on the development of adaptive, robust and generalized emotion recognition systems, we also spend lots of time and efforts on experiment design, data collection, and raw data preprocessing. They are the important and indispensable parts of this research work.Doctor of Philosophy (SCE