Modeling nonverbal behaviors for virtual agents

2012-01-06Virtual agents are autonomous software characters that support face-to-face interactions with human users. They are capable of understanding human input (i.e. speech, text input) and automatically generating responses that are adaptive to the context of the interaction. By communicating through verbal and nonverbal channels (i.e. behaviors without words), virtual agents can support meaningful social interactions with human users. ❧ One of the main goals in virtual agent research is to emulate how humans interact face-to-face. While traditional human-agent interaction was mainly accomplished through speech or text, with the emergence of better graphical representation and control over the virtual agent's embodiment, communication through nonverbal behaviors has become an active research area. However, nonverbal behavior is complex, involving many different behaviors, such as facial expressions, arm gestures, and gaze movements. There is also a complex mapping between nonverbal behaviors and their impact on communication (the communicative functions) and this creates a great challenge in the task of behavior authoring for virtual agents. ❧ The central goal of this research is to explore ways to derive models that automatically generate nonverbal behaviors and can thereby greatly facilitate behavior authoring. In this research, two major approaches to provide computational frameworks for generating nonverbal behaviors are explored: a literature-based approach and a machine learning approach. The former approach encodes the findings of psychological research into a set of rules, which is validated and prioritized through additional video analysis. The framework developed from this work has been incorporated within a growing number of different virtual agent systems. The machine learning approach focuses on learning the patterns of speaker behaviors, including head nods and eyebrow movements, as well as the behaviors of people with different characteristics (i.e. different job roles and behavioral traits). The objective evaluations show that the probabilistic models learned on a subgroup of people achieved better learning performance and the subjective evaluation study shows that the characteristics of the subgroups of people learned by the models can be carried through their generated behaviors. ❧ This research contributes to the development of virtual agents by facilitating the behavior authoring process, providing comparisons of different approaches for modeling nonverbal behaviors that has not been studied extensively before, and understanding the discrepancies between the behavior modeling process and the perception of the behaviors through evaluation studies with human subjects