Multimodal Communication from Multimodal Thinking - Towards an Integrated Model of Speech and Gesture Production

Kopp S, Bergmann K, Wachsmuth I. Multimodal Communication from Multimodal Thinking - Towards an Integrated Model of Speech and Gesture Production. Semantic Computing. 2008;2(01):115-136.A computational model for the automatic production of combined speech and iconic gesture is presented. The generation of multimodal behavior is grounded in processes of multimodal thinking, in which a propositional representation interacts and interfaces with an imagistic representation of visuo-spatial imagery. An integrated architecture for this is described, in which the planning of content and the planning of form across both modalities proceed in an interactive manner. Results from an empirical study are reported that inform the on-the-spot formation of gestures