Haptic Exploration Behavior During Bimodal Object Recognition

For the purpose of object recognition, the combination of complementary information derived from the different sensory systems (vision and touch) should result in a rich representation of the object in memory and may consequently enhance recognition performance. The purpose of this study is to investigate the haptic exploration behavior during bimodal object recognition. Specifically, we asked whether the exploration behavior for recognition depends on what modality was used during learning of an object. Analog to [1], we designed an old/new object recognition task using novel objects built each from 6 Lego bricks. In each condition subjects had to learn 4 novel object shapes either haptically (H), visually (V), or combined haptically and visually (VH). In a following recognition phase we added 4 distractor objects and subjects had to decide whether the presented object was known (old) or unknown (new) from the learning phase. The recognition phase was always bimodal (VH). We tested 12 subjects in all three conditions (V-to-VH, H-to-VH, VH-to-VH) measuring recognition performance. Subjects exploration behavior was recorded using video tape. For technical reasons, the recordings of only 8 subjects was used for the video analysis. Therefore, the video was cut into single clips each showing the exploration of one stimulus. These clips were replayed in randomized order to raters, who judged the ‘hapticallity’ of subjects’ exploration behavior on a scale with 9 possible answers between ‘subject just held and turned the stimulus’ to ‘subject explored the stimulus with fingers’. We found ‘hapticallity’ during visualhaptic recognition was judged largest for the exploration following haptic learning (H-to-VH), smallest for the one following visual learning (V-to-VH), and intermediate for visuo-haptic learning (VH-to-VH). This suggests that subjects use a strategy when recognizing the objects in a way that they match the exploration behavior between learning and recognition test. They use the haptic modality for recognition when the object was learned haptically. When the object was learned visually they predominantly use the visual modality during recognition ignoring touch. Also when the object was learned bimodally (VH), the exploration strategy was matched between learning and recognition test here indicating that the haptic modality was used almost exclusively for actively manipulating the object but not for exploring its shape