Modeling image interpretation through integration of geometric and photometric representations

Existing computer vision systems continue to have difficulty performing basic tasks in complex scenes despite decades of research. Much of this difficulty results from incomplete understanding of the relationships between observed intensity image data, extracted segmentation data, and semantic or global models used to represent and reason about objects and scene content. We address this knowledge gap by analyzing interactions between scene content and the reliability of image interpretation. Given images of a scene and prior information about the scene, such as coarse geometry, surface albedo and viewpoint, we compare observed data to analytically predicted appearance over a set of scene and image conditions. From this analysis, we postulate that analytical appearance prediction is insufficient in many problem domains, largely because of segmentation errors and modeling inaccuracy. Hence, we define a geometric, 3D object representation that maps learned appearance information onto geometric features to provide a global model incorporating the local relationships between scene content and observed appearance. Based on the aspect graph and parameterized by viewpoint and illumination orientation, the “interpretation model” is validated empirically through statistical analysis on real aerial images. The predictive ability of the interpretation model is shown to compare favorably with analytical appearance prediction on complex scenes. Methods of computing quantitative match criteria between the interpretation model and previously unseen imagery are derived to evaluate the quality of interpretation. The representation is then used to assess the tradeoff between prior and learned information required to achieve a desired level of image interpretation accuracy. Its utility in improving the robustness of high-level algorithms is demonstrated by applying the representation to pose adjustment and change detection