Recognition of Masked Facial Expressions Based on Convolutional Neural Networks and Data Augmentation

People sometimes intentionally change their facial expressions to mask their real emotions. Currently, there is relatively little research on the recognition of masked expressions. The effective identification of masked expressions helps to identify deception and lies and has important application value. Traditional machine learning methods require the manual design of feature extraction algorithms and have poor performance in facial expression recognition. This study performs masked expression recognition based on convolutional neural networks, including VGGNet, GoogLeNet, ResNet, and MobileNet. Data augmentation and regularization are used to improve the performance of the network. The recognition rate of masked expressions has been significantly improved. Using leave-one-subject-out cross-validation, the recognition accuracy is 63.62% for required expressions (6R), 39.97% for experienced emotions evoked by video clips (6E), and 22.11% for 36 mixed expressions (6Ex6R), which is improved by 33.19%, 48.99%, and 96.52% compared to traditional methods, respectively.