Speech capable Customer Service Bot with Video-Audio based Emotion Recognition

In today’s world, feedback from the customer is one of the essential aspects to improve any business. It is an added advantage to understand the customer emotion along with the feedback. This research aims to recognize facial emotion from the user and simultaneously capture the emotion through vocal Text. Two datasets were used in this research, FER2013, a dataset containing human faces classified to different emotions. The other is Dailydialogue, which contains day-day speech of humans and corresponding emotions. Initially, Convolution Neural Network was chosen for facial emotion recognition. The accuracy of the CNN model was comparatively low. Hence, to improve the model’s performance, the Capsule network is combined with CNN, and the CNN-CapsNet model is implemented. Naive Bayes algorithm is used for the sentimental analysis of Text, converting from speech to Text, and corresponding emotion is detected. The desired output of both the implemented models is the user’s emotion, which is then merged using weighted sum probabilities. For validation of the entire process flow, the model is implemented into a real-time chatter audio bot. The initial accuracy of the CNN model was around 65%, which is then improved to 89.5% using the CNN-CapsNet model for more accurate recognition