A Real-Time User Interest Meter Based on Human Cognitive Model and its Applications

在使用者與3D魔幻水晶球(將虛擬的3D文物呈現在透明的水晶球內，並且讓使用者可以直接用手在水晶球上操作虛擬的文物)的互動過程中，我們觀察到使用者有許多有趣的反應，我們想將每位使用者有趣的反應收集起來並彙整成個人的專屬回憶。因此我們提出一個以多個人類認知模型為基礎的“興趣量測系統”來讓電腦更了解使用者的反應，每當使用者與電腦互動時，興趣量測系統會即時地根據使用者的反應來量測使用者的興趣程度。其中，使用者反應包含了眼球的移動、眨眼、頭部的移動以及臉部的表情。另外我們提出一個實驗來證實興趣量測系統的確能夠量測出使用者的興趣程度。最後會舉出兩個整合範例，第一是在3D魔幻水晶球互動系統中依據使用者的反應將有趣的畫面整合為使用者的專屬回憶，第二是將使用者在觀看自己所拍攝的影片時的反應資訊加入影片剪輯系統中並自動剪輯出MV型式的精華影片。我們的應用及實驗證實了本論文所提出的性趣量測系統的確可以量測使用者的興趣，並且在應用中激發出更合適的互動模式。In users’ interaction process of the 3D Magic Crystal Ball, an interactive visual display system which allows users to see a 3D virtual artifact appearing inside a transparent glass ball and to manipulate it with bared hands, we find that there are many interesting reactions of users. We want to collect each user’s interesting reactions and combine them into personal memory. Therefore, we propose the Interest Meter, a system making computer understand user’s reactions, based on multimodal interfaces for measuring user’s interest in real time. The Interest Meter takes account of users’ spontaneous reactions when users interact with computers. In this work, we analyze the variations of user’s eye movement, blinking, head motion, and facial expression when the user interacts with computers. Furthermore, we propose the method of combining those signals into interest level and verify that it works in our experiment. There are two integrated application in our thesis. First, produces the personal memory of each user according to the user’s reactions to the Magic Crystal Ball. Second, edits the MV-style home video auto-matically by the user’s reactions during watching home videos. According to our experiments and applications, it shows that the Interest Meter can measure user’s interest and make a great improvement of the interac-tion.1 Introduction 1 Related Work 5.1 Unimodal Interface V.S Multimodal Interface 5.2 Multimodality in Emotion Recognition 6.3 Real-Time Affective Multimodal Interactive Applications 7 System Framework 12 Interest Meter 15.1 Attention Model 15.1.1 Head Motion Detection 16.1.2 Blinking Detection 17.1.3 Saccade Detection 18.1.4 Attention Score Computing 19.2 Emotion Model 20.2.1 Facial Expression Recognition 21.2.2 Emotion Score Computing 26.3 Information Fusion 27.3.1 Interest Score Computing 29.3.2 Weighting Adjustment 29.4 Experiments 32 Applications 37.1 Magic Crystal Ball 37.2 MV-Style Home Video Automatic Editing System 41 Conclusion and Future Work 43.1 Conclusion 43.2 Future Work 43ibliography 4