Machine learning and sentient embodied conversational agents: design and implementation of a virtual tutor in the context of energy efficiency and sustainability

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2018, Director: Pablo Almajano[eng] Current increase of chatbots’ popularity in the Human Computer Interaction field has lead to a proliferation of platforms that facilitate their design and seamless integration. However, chatbots are non-embodied conversational agents designed for communicating with the user using simple and quick goal-based interactions that fall short when it comes to engage users in longer, diverse and complex conversations. This work presents Sentient Embodied Conversational Agents (SECAs), which are embodied virtual characters. SECAs can engage users in complex structured conversations and also incorporate some human-like sentient qualities like being able to perceive user’s feelings and response to them. This project includes the formalization of a SECA architecture and the implementation of a software library that facilitates the inclusion of SECAs in applications requiring proactive and sensitive agent behaviours. SECA library has a modular design to facilitate its modification. It includes a controller and different modules to model different features SECAs have (conversational capability, knowledge, memory, personality, needs and empathy). Since educational applications constitute an example with such requirements, this work designs, implements and evaluates a virtual tutor (the Earth), which uses the presented architecture and software library, and embeds it in a gamified cultural probes application for children. This SECA’s purpose is enhancing the user experience and making it more educative. This project performs two complete iterations of the User Centered Design methodology process. The first one embeds in the application a first prototype of the agent using keyword-pattern matching techniques to analyse user input. On the other hand, the second one uses user feedback and data gathered to create an enhanced version of the agent with Machine Learning. The final version of the software library includes a Natural Language Processing Module integrating keyword-pattern matching techniques and ML with the purpose of improving SECAs understandability. Evaluation results of the application with the Earth SECA embedded show that children are satisfied both in terms of their perception of learning and overall experience. Additionally, there are significant differences in quantity of user data gathered by the new CP application (with SECA) and its previous version (without SECA). Though there is still room for improvement, NLP Module also improved the user input analysis results with respect of the initial prototype of the application with the Earth SECA using only keyword-pattern matching techniques