Orientation fields in the perception of 3D shape

If you pick up a typical textbook on perception you'll learn that there are many cues to 3D shape, such as texture, shading, highlights, perspective, etc. Each of these sources of information has a different physical cause. Thus, to interpret each cue, the visual system must impose a different set of computational constraints. This has lead to the widely held belief that each 3D shape cue is processed by a separate, dedicated processing stream or ‘module’. A considerable amount of research has gone into working out how accurate shape estimates can be derived from each cue, and how these independent estimates can be combined optimally. However, surprisingly little work has been done to try and find commonalities between the various cues. Here we show theoretically how shape from shading, highlights, texture, perspective and possibly even stereo can share some common processing tricks. The key insight is that the projection of 3D surfaces into 2D images introduces dramatic local image anisotropies that depend directly on properties of the 3D shape. Globally, these anisotropies are organized into smooth, continuous, swirling patterns, which we call ‘orientation fields’. We have argued recently (Fleming, et al. JOV 4(9), 2004) that orientation fields can be used to recover shape from specularities. Here we show how orientation fields could play a role in a wider range of cues. For example, although diffuse shading looks completely unlike mirror reflections, in both cases there is a systematic mapping from 3D surface curvatures to 2D image gradients. Thus, both shading and specularities lead to similar orientation fields. The mapping from orientation fields to 3D shape is different for other cues, and we exploit this to create powerful illusions. We also show how some simple image-processing tricks could allow the visual system to ‘translate’ between cues. Finally, we outline the remaining problems that have to be solved to develop a ‘unified theory’ of 3D shape recovery