Three-dimensional modeling based on disparity estimation for image-based virtual environments

This thesis addresses the challenging problem of obtaining 3D models for real environments from stereo images and translational video sequences. The problem is partitioned into two main parts: matching and disparity estimations to obtain depth maps and separate 3D models for different image locations, and the combining of these separate 3D models into one 3D model for the whole environment. Solutions are proposed dealing with these two main issues respectively, and the results from implementing these solutions are also presented. The novelty for the solution of the first part -- which deals with the matching problem -- lies in the fact that it combines the pixel-based approach and region-based approach. A hybrid algorithm is developed for disparity estimation between stereo images and translational video sequences, so that some intrinsic problems in the pixel-based and region-based approaches (like the detection of sky) can be solved in a combined way while keeping the object boundaries sharp and crisp, and hence give disparity and depth maps which are qualitatively better than traditional methods. The novelty for the solution of the second part -- which deals with the integration of separate 3D models from different image locations -- comes from the usage of region information obtained from the disparity estimation process, the estimation of ego-motion parameters, as well as to the integration of the object surfaces from different 3D models. For the ego-motion estimation, instead of using the bundle adjustment or iterative closest points (ICP) which perform the estimation in 3D space, our algorithm uses large regions with correspondence information in each image to determine the homogeneous transformations in image space. For the integration of separate 3D models, these large regions from different models are also used to adjust and expand the shape of regions which belong to the same surfaces, so that even the occluded surfaces in one image location can be filled by integration of surfaces from other images. The results are shown with the image rendering at novel viewpoints as well as with PSNR values measured between the rendered images and existing images at real image locations