Grasping gestures: gesturing with physical artifacts

Gestures play an important role in communication. They support the listener, who is trying to understand the speaker. However, they also support the speaker by facilitating the conceptualization and verbalization of messages and reducing cognitive load. Gestures thus play an important role in collaboration and also in problem-solving tasks. In human–computer interaction, gestures are also used to facilitate communication with digital applications, because their expressive nature can enable less constraining and more intuitive digital interactions than conventional user interfaces. Although gesture research in the social sciences typically considers empty-handed gestures, digital gesture interactions often make use of hand-held objects or touch surfaces to capture gestures that would be difficult to track in free space. In most cases, the physical objects used to make these gestures serve primarily as a means of sensing or input. In contrast, tangible interaction makes use of physical objects as embodiments of digital information. The physical objects in a tangible interface thus serve as representations as well as controls for the digital information they are associated with. Building on this concept, gesture interaction has the potential to make use of the physical properties of hand-held objects to enhance or change the functionality of the gestures made. In this paper, we look at the design opportunities that arise at the intersection of gesture and tangible interaction. We believe that gesturing while holding physical artifacts opens up a new interaction design space for collaborative digital applications that is largely unexplored. We provide a survey of gesture interaction work as it relates to tangible and touch interaction. Based on this survey, we define the design space of tangible gesture interaction as the use of physical devices for facilitating, supporting, enhancing, or tracking gestures people make for digital interaction purposes, and outline the design opportunities in this space