Multimodal contact cues for object manipulation in augmented and virtual environments

This thesis aims to increase our knowledge about the planning and organization of aiming and prehensile movements in augmented and virtual environments and to better understand how sensory information, such as vision, haptics and especially audition, is used for planning and control of simple movements. Five experiments were performed to investigate the importance of sensory contact cues when pointing and grasping augmented and virtual targets in computer-generated environments, with the hand and with a tool. Kinematic characteristics of transport and grasp movements as well as spatial error measures were examined. Experiment 1 explored the roles of natural haptic and computer-generated auditory contact information when grasping target objects of four different sizes. Results revealed the importance of contact information, and the benefit of auditory contact cues for improving reach to grasp performance in augmented and particularly in virtual environments. In Experiment 2 we introduced graphic contact cues, presented alone and in combination with auditory and haptic contact cues for a grasping task. We investigated the potential benefits and limitations of providing contact information in multiple sensory modalities. Overall, results were similar for auditory and auditory plus graphic conditions, while the graphic only condition mostly resembled the no cue condition. Experiment 3 extended the study of computer-generated auditory contact cues as redundant and substitutive for natural haptic information when grasping with a tool and found benefits of auditory cues on both kinematics and spatial error measures. Experiments 4 and 5 further investigated the effects of auditory and haptic contact cues on a pointing task, with the hand and with a tool. Both natural haptic and auditory contact cues had significant effects on kinematics and spatial error measures. Results are discussed in terms of implications for understanding multimodal information processing during manipulation, and for the design of multimodal displays in augmented and virtual environments