Development and evaluation of a system for automatic emotion recognition from facial expressions

Učinkovitost interakcije med človekom in strojem je odvisna tudi od tega, kako relevantne podatke ima stroj o človeku. S prihodom vse bolj avtonomnih strojev se pojavlja potreba po tem, da bi stroj lahko sam zaznaval čustveno stanje človeka. V tem magistrskem delu smo se osredotočili na pristop prepoznavanja čustev na daljavo s pomočjo spletne kamere. Izdelali smo lasten sistem za prepoznavo čustev na podlagi obrazne mimike, ki temelji na modernem orodju OpenFace za izračun značilnic (primernih kot vhod metodi strojnega učenja) in za klasifikacijo čustev uporablja metodo podpornih vektorjev (SVM). Svoj pristop smo primerjali s priznanim sistemom Noldus FaceReader. Uporabili smo metrike natančnosti, priklica, točnosti ter F1, ki predstavlja celovitejšo oceno klasifikatorja. Za ocenjevanje algoritmov smo uporabili naslednje zbirke videoposnetkov oz. slik: (1) Bahcesehir University Multimodal Face Database of Affective and Mental States (BAUM-1) in (2) Geneva Multimodal Emotion Portrayals Core Set (GEMEP), katerih posnetki so nastali s snemanjem obrazov udeležencev posameznih raziskav ob induciranju določenih čustvenih stanj, ter (3) zbirko slik s kategoričnimi oznakami, ki smo jih lastnoročno izbrali iz zbirke BAUM-1s. Zbirki videoposnetkov vsebujeta tako igrane kot tudi pristne odzive na dražljaje. Primerjali smo rezultate klasifikatorjev ob različnih načinih interpretacije klasifikacij, nastale razlike analizirali glede na zbirke vhodnih podatkov ter izpostavili vrsto faktorjev, ki so vplivali na rezultate. FaceReader se je pri klasifikaciji izkazal za bolj konsistentnega, v povprečju pa je naš klasifikator dosegel boljši rezultat F1. Vsak klasifikator ima svoje prednosti in slabosti. V diskusiji smo se poleg obravnave slednjih dotaknili tudi možnih izboljšav. Dovolj visoka uspešnost metod za merjenje čustev bi omogočila obstoj vrste aplikacij s koristnimi zmožnostmi za mnoga področja, kot so varnost, nega ljudi, administracija podjetij in držav itd.For a machine to have efficient human-computer interaction, relevant information about the subject is needed. As the automation of an increasing number of tasks is being developed, the need for artificial emotion recognition solutions is becoming more apparent. In this work, we focus on the solutions that can measure human emotions from a distance by using a camera. We developed one that uses detected facial expressions to classify emotions. It employs OpenFace to measure present facial action units which are in turn used as the input to a Support Vector Machine classifier. We tested our method’s performance against Noldus FaceReader by using metrics of precision, recall, accuracy, and F1 score. For testing, we used two multi-modal datasets with emotion-annotated video files: (1) Bahcesehir University Multimodal Face Database of Affective and Mental States (BAUM-1, which has two parts – BAUM-1s and BAUM-1a) and (2) Geneva Multimodal Emotion Portrayals Core Set (GEMEP)we also added a dataset of emotion-annotated images that we compiled by manual selection from the BAUM-1s database. While GEMEP and BAUM-1a databases contain acted emotional expressions, BAUM-1s is only composed of videos with spontaneous emotional expressions. We compared the classifiers’ performance while using different interpretations of probabilistic classifications, analyzed the differences based on the used input datasets, and discussed factors responsible for the measured outcomes. While FaceReader seemed to perform more consistently, our classification method achieved a better mean in F1 scores, even though the difference was not statistically significant. The pros and cons of each classifier and possible classifier upgrades are discussed. A high enough performance of such emotion-recognizing systems would enable the development of useful applications for various purposes, such as security, healthcare, administration, marketing, and many others