Improve OpenMVG and create a novel algorithm for novel view synthesis from point clouds

This thesis presents work to improve open source 3D reconstruction software OpenMVG and to create a novel algorithm to render photorealistic images from new views given a photo collection and 3D point cloud. First, the original OpenMVG is parallelized using GPU and its data structure is optimized. Moreover, we integrated the MatchMiner algorithm into OpenMVG to further improve its efficiency. Last but not least, an initial pair selection formulation and a default focal length setting are introduced and implemented to automize OpenMVG. Then 3D sparse point clouds of construction sites are reconstructed by performing Structure-from-Motion (SfM) with the improved version of OpenMVG and source images (images that are used in SfM) are calibrated and registered to point clouds. Furukawa's Patch-based Multi-view Stereo(PMVS) algorithm is used to reconstruct dense point clouds using calibrated cameras as inputs. With known depth values of 3D points in the dense point cloud, we estimate depth maps of source images using optimization similar to Levin's colorization algorithm. For a novel view of the point cloud, we find source images that share some common elements of the construction site that are also visible to the novel view. Then we warp depth maps of these candidate images to the novel view. We estimate a depth map and label pixels for the novel view by solving a multi-label Markov Random Field (MRF) optimization problem using graph-cuts. We introduce a novel energy minimization formulation exploits both 2D and 3D information. Finally, a photorealistic image of the novel view is rendered by copying pixel colors from selected candidate source images using pixel labels computed with graph-cuts. We experimentally validate our approach on several challenging viewing angles of a point cloud model of a complicate construction site. The rendered results show high photo-realistic synthesis quality in planar scenes