Holistic Video Stitching for Street Panorama

Coordinated Science Laboratory was formerly known as Control Systems LaboratoryIn this paper, we address how to automatically generate a panorama for a street view from a long video sequence. We model the panorama as a low-rank matrix and formulate the problem as one of robust recovery of the low-rank matrix from highly incomplete, corrupted, deformed measurements (the video frames). We leverage powerful high-dimensional convex optimization tools from compressive sensing of sparse signals and low-rank matrices to solve this problem. In particular, we show how the new method can effectively remove severe occlusions or corruptions (caused by trees, cars, or reflections, etc.), and obtain clean, intrinsic street panoramas that are consistent with all frames. We also show how our method can automatically and robustly establish pixel-wise accurate registration among all the video frames. We demonstrate the effectiveness of our method by conducting extensive experimental comparison with other popular video stitching methods such as AutoStitch and Adobe Photoshop.National Science Foundation / NSF IIS 11-1601