Integration of simultaneous visual and haptic information

When a person looks at an object while exploring it with their hand, vision and touch both provide useful information for estimating the properties of the object. Here, we investigated the fusion of visual and haptic information and its limits. We propose that a general principle, which minimizes variance in the final estimate, determines the degree to which vision or haptics determines the integrated, visual-haptic percept. This principle is realized by using maximum-likelihood estimation (MLE) to combine the inputs. To investigate cue combination quantitatively, we first measured the variances associated with visual and haptic estimation of height. Those measurements were then used to construct the MLE integrator. The model and humans behaved very similarly in a visual-haptic height discrimination task. Thus, the nervous system seems to combine visual and haptic height information in a fashion quite similar to MLE integration. In a second study we used an oddity task to investigate the break down of visual-haptic fusion. Three horizontal bars were presented sequentially. Two of them were identical and had equal visual and haptic heights (standard stimulus); the third had a visual and/or haptic height differing from the standard (odd stimulus). Subjects indicated which of the three intervals contained the odd stimulus. If subjects relied on the fused visual-haptic estimate, discrimination should be most difficult when the weighted average of the visual and haptic heights corresponds to the standards height and easiest if both heights of the odd stimulus are either bigger or smaller then the standards height. In contrast, if subjects used the visual or haptic information independently without fusing them, discrimination would occur whenever either the visual or the haptic height in the odd stimulus differed noticeably from the standards height. We found that discrimination was indeed most difficult when the weighted averages were the same and easiest when they differed. Thus, the fused visual-haptic percept is used for discriminating between the stimuli. However, if the conflict between the visual and haptic stimuli became too large, this difference in discrimination performance was not observed anymore. In other words, visual-haptic fusion breaks with large conflicts. Remarkably, we observed metameric behavior in some condition. That is, discrimination would be better if subjects simply shut their eyes or removed their hand from the bar instead of using the fused visual-haptic information