Depth super resolution by rigid body self-similarity in 3d

We tackle the problem of jointly increasing the spatial resolution and apparent measurement accuracy of an input low-resolution, noisy, and perhaps heavily quantized depth map. In stark contrast to earlier work, we make no use of ancillary data like a color image at the target resolu-tion, multiple aligned depth maps, or a database of high-resolution depth exemplars. Instead, we proceed by identi-fying and merging patch correspondences within the input depth map itself, exploiting patchwise scene self-similarity across depth such as repetition of geometric primitives or object symmetry. While the notion of ‘single-image ’ super resolution has successfully been applied in the context of color and intensity images, we are to our knowledge the first to present a tailored analogue for depth images. Rather than reason in terms of patches of 2D pixels as others have before us, our key contribution is to proceed by reason-ing in terms of patches of 3D points, with matched patch pairs related by a respective 6 DoF rigid body motion in 3D. In support of obtaining a dense correspondence field in reasonable time, we introduce a new 3D variant of Patch-Match. A third contribution is a simple, yet effective patch upscaling and merging technique, which predicts sharp ob-ject boundaries at the target resolution. We show that our results are highly competitive with those of alternative tech-niques leveraging even a color image at the target resolu-tion or a database of high-resolution depth exemplars. 1