Modeling Interpersonal Coordination and Grounding Behavior in Joint Activities with Social Companions

Interpersonal coordination and grounding are fascinating phenomena that underlie all natural social interactions and everyday joint activities between humans. Interpersonal coordination takes place whenever people are smoothly adapting to each other's interaction tempo and rhythm, or are seamlessly synchronizing and intertwining their actions and behaviors. Grounding denotes all collaborative efforts that are involved in establishing, maintaining, and repairing the common perceptual and conversational ground during a joint activity. Interpersonal coordination and grounding are influenced by a variety of highly interwoven behavioral functions that have different social and regulative impacts and include numerous information modalities. For example, various gaze behaviors are involved in attention following, multi-modal disambiguation, turn management, feedback generation, intimacy regulation, and personality evaluation. Another important aspect is speech overlaps and interruption attempts that have a significant effect on the interaction flow as well as interpersonal attitudes, such as involvement, dominance, and affiliation. Mastering the close coordination of the above functions and their interplay with the dialog management plays an important role for interpersonal coordination and grounding in physically situated joint activities. This mainly comprises the proper prioritization and tight synchronization of the underlying incremental, reciprocal, and concurrent behavioral processes. While this seems to work so well and intuitively in natural human interactions, already from birth on, it obviously works by no means as perfectly in social human-agent interactions. Smallest discrepancies in the synchronization or prioritization of individual behavioral aspects may already make a social companion appear unnatural, incompetent, or clumsy. For that reason, the close coordination of the behavioral functions and the underlying behavioral processes is the major challenge in modeling the behavior and interaction of a social agent. In order to meet this modeling challenge, an author must cope with three modeling tasks, each of which establishing task-specific requirements for an sufficiently expressive and practicable modeling approach. The first task involves the creation of versatile, multi-modal behavior compositions that integrate context knowledge and can automatically be varied. The second task includes the evaluation of temporal and semantic integration constraints as well as various quantification operations for multi-modal fusion and knowledge reasoning. The last task comprises the proper incremental interleaving of behavior recognition, knowledge reasoning and behavior generation as well as the close coordination, prioritization, interruption, and resumption of concurrent, nested, and intertwined processes underlying the various behavioral functions on different levels of the behavior model. A review of state-of-the-art related efforts shows that, so far, there has not been presented a uniform behavior and interaction modeling approach for social companions that masters all of these tasks and requirements. Some related work has studied and modeled individual behavioral functions or aspects in isolation without considering their complex interplay with other functions or their role for interpersonal coordination and grounding. Other research has focused on the design of general purpose modeling languages for only one individual modeling tasks but misses to demonstrate how to interlink their solution with the other tasks and how to use them for modeling interpersonal coordination and grounding behaviors. This thesis presents a novel modeling approach for the interactive behavior of artificially and socially intelligent agents that is precisely designed to meet the above requirements and overcome the shortcomings of related work. It is the first approach to combine a specially designed, hierarchical and concurrent state-chart variant, a domain-specific, logic, multi-modal fusion and reasoning calculus, and a template-based behavior description language into a uniform behavior and interaction modeling framework. The proposed approach significantly facilitates the distributed and iterative development of clearly structured, easily adaptable, extensible, and reusable computational dialog, behavior, and interaction models of social agents. While being sufficiently expressive, it is remarkably practicable since it primarily relies on visual and declarative modeling formalisms. It is suitable for rapid-prototyping and the creation of sophisticated models by differently experienced authors