Combining sensory information: mandatory fusion within, but not between, senses

Humans usemultiple sources of sensory information to estimate environmental properties. For example, the eyes and hands both provide relevant information about an object’s shape. The eyes estimate shape using binocular disparity, perspective projection, etc. The hands supply haptic shape information by means of tactile and proprioceptive cues. Combining information across cues can improve estimation of object properties but may come at a cost: loss of single-cue information. We report that single-cue information is indeed lost when cues from within the same sensory modality (disparity and texture gradients in vision) are combined, but not when different modalities (vision and haptics) are combined. Sensory estimates of an environmental prop-erty can be represented by Ŝi fi(S) where S is the physical property being estimated, f is the operation the nervous system performs to derive the estimate, and S ̂ is the perceptual estimate. The subscripts refer to different sen-sory modalities (e.g., haptics and vision) or different cues within a modality (e.g., dispar-ity and texture cues within vision). Sensory estimates are subject to two types of error: random measurement error and bias. Thus, estimates of the same object property from different cues usually differ. To reconcile the discrepancy, the nervous system must either combine estimates or choose one, thereby ignoring the other cues. Assuming that each single-cue estimate is unbiased (1) but cor-rupted by independent Gaussian noise, the statistically optimal strategy for cue combi-nation is a weighted average (2, 3