Automatic recognition of human behaviour in sequential data

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonAutomatic human behaviour recognition is a very important element for intelligent Human Computer Interaction (HCI) in which the machine or computer can recognize human behaviour and respond to humans accordingly. Among human behaviour recognition tasks, dynamics represents key information of the action, and it is one of the hardest tasks for automatic recognition. Significant progress has been made in the Artificial Intelligence (A.I.) area recently that provides new tools and technologies to recognize visual objects and make better decisions, logical deductions, mathematical optimization, etc. In this thesis, A.I. technologies have been applied to the analysis of human behaviour, especially using dynamic clues for touch gestures in which a human touches an animal or an object, etc.; micro-gestures that can be extensively utilized on wearable devices; body gestures that are not only used as a paralanguage, but also as an indicator of body behaviour; and 3D facial expression analysis that extracts the emotion information from high quality high-resolution 3D video recordings. Firstly, an automatic touch gesture recognition system has been proposed, including preprocessing, multiple feature extraction, feature selection, pattern recognition and fusion. Both statistical and video features were extracted, including Motion Statistical Distribution (MSD), Spatial Multi-scale Motion History Histogram (SMMHH), Binary Motion History (BMH), Statistical Distribution (SD) and Local Binary Pattern on Three Orthogonal Planes (LBPTOP). Two powerful machine learning methods, Random Forest and multiboosting, have been utilized. A Sobel edge detection is utilized as pre-processing, and a Minimum Redundancy and Maximum Relevance (mRMR) feature selection is used to reduce the dimension of features after feature extraction. A decision-level fusion method Hierarchical Committee (HC) has been used as a post-processing tool to combine all the predictions. The main contribution of the system is the versatility of it, which can be applied in different dataset. This system also achieves a high performance with maintaining the versatility. Secondly, another automatic 3D micro-gesture recognition system has been proposed and tested on a Holoscopic Micro 3D Gesture (HoMG) dataset for which a holoscopic 3D video was recorded and annotated. A new system including frame selection by score has been proposed on the video-based dataset. Video-based recognition used LBPTOP and the Local Phase Quantisation from Three Orthogonal Planes (LPQTOP) as feature selection and Support Vector Machine (SVM) as machine learning. Then an SVM prediction has been utilized, and a score of each frame has been predicted. After using the SVM score to reduce the frames on the video-based dataset, the performance of video-based recognition is improved. This 3D micro-gesture recognition system achieves the best performance comparing with other current works by considering the non-linear relationship of features. Thirdly, an automatic body gesture recognition system has been proposed to help older people with Chronic Lower Back Pain (CLBP). The proposed system can recognize the behaviours of CLBP patients, like abrupt actions and guarding. A new two-stage machine learning method has been proposed that combines k-nearest neighbour k-NN and HMM and achieves a better recognition performance of body gestures than traditional methods. The contribution of the system is it could detectt and analysis CLBP related body behaviour frame by frame and provide more detailed information about CLBP including the starting, ending and different level of CLBP according to the time series, which would help the future research of CLBP. Fourthly, a 3D feature expression recognition (FER) system has been proposed to achieve better performance on the most popular posed face 3D FER dataset, BU4D. A latest Background Subtraction method was applied based on tensor for pre-processing to extract the dynamic information on the face. This is the first utilized Tensor Background Subtraction From Compressive Measurements (BSCM) on FER. A deep learning method (e.g. Dynamic Image Net) is used on the dynamic facial information. A comparison between with and without Background Subtraction is made to prove the effectiveness of this method. The contribution of this system is providing a new solution of reduce the calculation resources and reduce the computing time. It also helps to remove noise from the original data and optimise the classification performance. Finally, a real-time FER system was built for an interactive movie system. In the proposed automatic emotion recognition system, CNN has been used as feature extraction, and SVM is used for classification. The real-time system has been prototyped and exhibited on several occasions. In the final system, several practical problems have been considered, including brightness control and the right face selection from all of the audience. That has improved the accuracy of the practical application. The contribution of the system is to achieve the aim of real-time FER while take care of the balance between system speed, classification accuracy and optimisation in real life using environment. In summary, several automatic recognition systems for human behaviour have been proposed and applied in real recording data and practical applications. Some methods have versatility, and some are specialized for distinct tasks. All these studies contribute to the development of intelligent HCI