A federated learning method for real-time emotion state classification from multi-modal streaming

Emotional and physical health are strongly connected and should be taken care of simultaneously to ensure completely healthy persons. A person’s emotional health can be determined by detecting emotional states from various physiological measurements (EDA, RB, EEG, etc.). Affective Computing has become the field of interest, which uses software and hardware to detect emotional states. In the IoT era, wearable sensor-based real-time multi-modal emotion state classification has become one of the hottest topics. In such setting, a data stream is generated from wearable-sensor devices, data accessibility is restricted to those devices only and usually a high data generation rate should be processed to achieve real-time emotion state responses. Additionally, protecting the users’ data privacy makes the processing of such data even more challenging. Traditional classifiers have limitations to achieve high accuracy of emotional state detection under demanding requirements of decentralized data and protecting users’ privacy of sensitive information as such classifiers need to see all data. Here comes the federated learning, whose main idea is to create a global classifier without accessing the users’ local data. Therefore, we have developed a federated learning framework for real-time emotion state classification using multi-modal physiological data streams from wearable sensors, called Fed-ReMECS. The main findings of our Fed-ReMECS framework are the development of an efficient and scalable real-time emotion classification system from distributed multimodal physiological data streams, where the global classifier is built without accessing (privacy protection) the users’ data in an IoT environment. The experimental study is conducted using the popularly used multi-modal benchmark DEAP dataset for emotion classification. The results show the effectiveness of our developed approach in terms of accuracy, efficiency, scalability and users’ data privacy protection.Arijit Nandi is a fellow of Eurecat’s “Vicente López” PhD grant program. This study has been partially funded by ACCIÓ Spain (Pla d’Actuació de Centres Tecnològics 2021) under the project TutorIA.Peer ReviewedPostprint (author's final draft