3D SCENE MODELING AND UNDERSTANDING FROM IMAGE SEQUENCES

Reconstructing and representing large-scale 3D scenes from image sequence have many important applications, including airborne or ground video surveillance for moving target extraction, automated 3D urban scene construction, airborne/ground traffic survey, and image-based modelling and rendering. To deal with two major challenges in large-scale 3D modelling- huge amount of data and intensive computation, we propose a dynamic mosaic-based approach that can model a large scale 3D scene with dynamic objects (Fig. 1a). The work in this thesis has the following four original contributions: • First, we extend the previous work on stereo mosaics from static scenes to dynamic scenes, thus allowing the handling of independent moving objects. This is significant in low-altitude aerial video surveillance for detecting moving targets (such as vehicles on roads) from an aerial platform. • Second, an effective and efficient patch-based stereo matching algorithm has been proposed to extract both 3D and motion information from stereo mosaics of urban scenes, which allows 3D reconstruction of scenes with a lot of texture-less regions with sharp depth boundaries (such as buildings). • Third, we perform thorough experimental analysis of the robustness and accuracy of 3D reconstruction using parallel-perspective stereo mosaics, showing the advantage of the parallel-perspective representation. • Fourth, a content-based 3D mosaic (CB3M) representation is proposed, and a graph-based higher-level scene understanding algorithm is proposed. The method shows the advantage of CB3M data representation in grouping planar patches into higher-level object entities, such as roads, buildings, etc