Modeling the semantic coordination of speech and gesture under cognitive and linguistic constraints

Bergmann K, Kahl S, Kopp S. Modeling the semantic coordination of speech and gesture under cognitive and linguistic constraints. In: Aylett R, Krenn B, Pelachaud C, Shimodaira H, eds. Intelligent Virtual Agents. Lecture Notes in Artificial Intelligence. Vol 8108. Berlin/Heidelberg: Springer; 2013: 203-216.This paper addresses the semantic coordination of speech and gesture, a major prerequisite when endowing virtual agents with convincing multimodal behavior. Previous research has focused on build- ing rule- or data-based models speci c for a particular language, culture or individual speaker, but without considering the underlying cognitive processes. We present a exible cognitive model in which both linguistic as well as cognitive constraints are considered in order to simulate natu- ral semantic coordination across speech and gesture. An implementation of this model is presented and rst simulation results, compatible with empirical data from the literature are reported