N.: Integrating semantics into multimodal interaction patterns

Abstract. A user experiment on multimodal interaction (speech, hand position and hand shapes) to study two major relationships: between the level of cognitive load experienced by users and the resulting multimodal interaction patterns; and how the semantics of the information being con-veyed affected those patterns. We found that as cognitive load increases, users ’ multimodal productions tend to become semantically more com-plementary and less redundant across modalities. This validates cognitive load theory as a theoretical background for understanding the occurrence of particular kinds of multimodal productions. Moreover, results indicate a significant relationship between the temporal multimodal integration pattern (7 patterns in this experiment) and the semantics of the com-mand being issued by the user (4 types of commands), shedding new light on previous research findings that assign a unique temporal inte-gration pattern to any given subject regardless of the communication taking place.