Shape From Textures: A Paradigm for Fusing Middle Level Vision Cues

This research proposes a new approach to the problem of deriving the orientation, segmentation, and classification of surfaces based on multiple independent textual cues. The generality of this approach is due to the interaction between textural cues, thus allowing it to extract shape information from a wider range of textured surfaces than any individual method. The method consists of three major phases: the calculation of orientation constraints for subimage elements called "texel patches", the consolidation of constraints into a "most likely" orientation per patch, and finally the reconstruction of the surface. During the first phase, the different shape-from-texture components generate augmented texels. Each augmented texel consists of the 2-D description of a texel patch and a list of weighted constraints on its orientation. The orientation constraints for each patch are potentially inconsistent or potentially incorrect because the shape-from methods are applied to noisy images, locally based, and derive constraints without a priori knowledge of the type of texture or number of surfaces. The constraints are weighted by each shape-from method based on an intra-cue correctness factor. This factor attempts to measure how closely the constraint fulfill the underlying assumptions of the cue. The orientation constraints' weights are then normalized between cues in order to assure that no cue predominates unfairly. In the second phase, all the orientation constraints for each augmented texel are consolidated into a single "most likely" orientation by a Hough-like transformation on a tesselated Gaussian sphere. The system iteratively reanalyzes each of the texel patches, calculating the "most likely" orientations for each patch. Finally, the system re-analyzes the orientation constraints to determine which augmented texels are part of the same constraint family and which cues were used to generated the valid constraints. In effect, this both segments the image into regions of similar orientation and supplies texture classification information. The robustness of this approach is illustrated by a system that fuses the orientation constraints of five shape-from cues and solves real camera-acquired imagery