Processing iconic gestures in a multimodal virtual construction environment

Fröhlich C, Biermann P, Latoschik ME, Wachsmuth I. Processing iconic gestures in a multimodal virtual construction environment. In: Bastos R, Wanderley MM, Gibet S, Dias M, eds. Gesture-Based Human-Computer Interaction and Simulation. Lecture Notes in Computer Science (LNAI). Vol 5085. Berlin: Springer; 2009: 187-192.In this paper we describe how coverbal iconic gestures can be used to express shape-related references to objects in a Virtual Construction Environment. Shape information is represented using Imagistic Description Trees (IDTs), an extended semantic representation which includes relational information (as well as numerical data) about the objects’ spatial features. The IDTs are generated online according to the trajectory of the user’s hand movements when the system is instructed to select an existing or to create a new object. A tight integration of the semantic information into the objects’ data structures allows to access this information via so-called semantic entities as interfaces during the multimodal analysis and integration process