Recovering Non-Rigid 3D Motion, Shape and Reflectance from Multi-View Image Sequences: A Differential-Geometric Approach

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2001. Simultaneously published in the Technical Report series.In this thesis we study the problem of recovering non-rigid motion, shape and reflectance properties of dynamic 3D scenes from image sequences. Our goals are both to advance towards a firmer mathematical understanding of the constraints that exist in this problem and to develop practical methods that extract the desired properties directly from visual data, using as little prior knowledge about the scenes being observed as possible. To recover motion, shape and reflectance simultaneously when they are all unknown and the scenes potentially have discontinuities, we observe that scenes composed of curves and surfaces with piecewise-smooth shape and motion trace manifolds embedded in 4D space-time as they move. Moreover, we show that these manifolds have a well-defined differential-geometric structure and, consequently, can be used as the basis to create spatiotemporally-distributed geometric and radiometric representations. This insight is supported by a mathematical analysis of how multi-view image sequences constrain spatiotemporally-localized scene properties such as the instantaneous 3D velocity, position and orientation of individual scene points. Based on this analysis, we develop a general framework for visual reconstruction of dynamic scenes, and propose specific representational primitives that are both powerful enough to capture a broad class of scenes with arbitrarily-high accuracy and simple enough to be unambiguously recovered from visual data alone. The use of these primitives leads us to develop algorithms that break the complex problem of reconstructing entire dynamic scenes into collections of spatiotemporally-localized, well-posed optimization problems. Experiments with complex real scenes (paper, clothing, skin, shiny objects) and scenes for which ground-truth geometry is known illustrate our methods' ability to (1) explain pixels and pixel variations in terms of their underlying physical causes---3D shape, surface reflectance, 3D motion, illumination, and visibility, (2) recover dense and non-rigid instantaneous velocity fields even in the presence of moving specularities, and (3) incorporate spatio-temporal coherence into computations for improved stability, and accuracy gains with respect to static multi-view analysis techniques