Estimation of multimedia exposure based on psychophysiological signals using of machine learning

Cilj magistrskega dela je izbor strojnega učenja za oceno učinkov oglasov na ljudi in ovrednotenje njegove uspešnosti. Z strojnim učenjem smo naučili model, ki na podlagi izbranih psihofizioloških signalov napoveduje razred dimenzije reaktance lestvice MMES. Dimenzija zajema negativni del čustev, ki jih uporabniku sprožijo oglasi. Signala, ki ju uporabimo za to oceno uporabimo, sta premer zenice v odvisnosti od časa in elektrodermalna aktivnost kože v odvisnosti od časa. V prvem delu magistrske naloge podamo teoretično ozadje ocen multimedijske izpostavljenosti (vprašalniki, s katerimi so določene itd.) ter ozadje psihofizioloških signalov. Nadaljujemo z razlago konceptov in algoritmov strojnega učenja, ki jih pri nalogi uporabimo. Posebno pozornost namenimo globokemu učenju. V raziskovalnem delu magistrske naloge najprej pred-obdelamo signale, predstavimo značilke signala in vizualiziramo podatke. Vizualiziramo porazdelitve značilk psihofizioloških signalov in porazdelitve značilk psihofizioloških signalov v odvisnosti od vrednosti reaktance. V nadaljevanju obdelan signal, značilke signala in tudi neobdelan signal uporabimo za učenje več različnih modelov strojnega učenja. Neobdelan signal uporabimo zgolj pri globokem učenju z nevronsko mrežo, izluščene značilke signala pa tudi pri drugih metodah strojnega učena. Rezultate ovrednotimo z mero F1 in ploščino pod ROC krivuljo. Na podlagi teh mer uspešnosti modele strojnega učenja primerjamo med seboj. Ugotovimo, da z modelom naključnih gozdov in značilkami obeh signalov, pri čemer je signal diametra zenice normaliziran, lahko bolje od naključja razvrstimo dimenzijo lestvice MMES reaktanco.The aim of this master thesis is to automate the assessment of the effects of advertisements on people. Using machine learning, we have trained a model that predicts reactance dimension on MMES scale based on selected psychophysiological signals. This dimension covers negative feelings caused by advertisements. Signals used are time series of pupil size and electrodermal activity. In the first part of the master thesis, we give the theoretical background of multimedia exposure estimation (questionnaires used to determine them etc.) and psychophysiological signals. We go on to explain the machine learning concepts and algorithms used in this thesis. Special attention is paid to deep learning. In the research part of the master thesis, we first process the signals, present the signal features, and visualise the data. We visualise the distributions of the psychophysiological signal features and the distributions of the psychophysiological signal features in relation to reactance. In the next part we use the processed signal, the signal features as well as the raw signal to train several machine learning models. The raw signal is only used in deep neural network learning, while the extracted signal features are also used in other machine learning methods. We evaluate the results using F1 scores and areas under the ROC curves. Based on these metrics, we compare the models with each other. We find that the random forest model and the features of both signals, with the pupil diameter signal normalized, can classify the MMES scale dimension reactance better than chance