Rapid creation of photorealistic virtual reality content with consumer depth cameras

2019-04-23High‐fidelity virtual content is essential for the creation of compelling and effective virtual reality (VR) experiences. However, creating photorealistic content is not easy, and handcrafting detailed 3D models can be time and labor intensive. Structured camera arrays, such as light‐stages, can scan and reconstruct high‐fidelity virtual models, but the expense makes this technology impractical for most users. ❧ In this dissertation, we first present a complete end‐to‐end pipeline for the capture, processing, and rendering of view‐dependent 3D models in virtual reality from a single consumer‐grade depth camera. To achieve photorealistic results, view‐dependent texture mapping (VDTM) method is used for realtime rendering to preserve the specular reflections and light‐burst effects. To further improve the overall visual quality, we then present a new VDTM approach, dynamic omnidirectional texture synthesis (DOTS), that synthesize a high‐resolution view‐dependent texture map for any virtual camera location. Synthetic textures are generated by uniformly sampling a spherical virtual camera set surrounding the virtual object, thereby enabling efficient real‐time rendering for all potential viewing directions. The results showed that DOTS were rated as superior visual quality compared to fixed textures and traditional VDTM, with DOTS producing smoother dynamic transitions between viewpoints. Although DOTS can produce promising texture at all viewpoints, post‐editing such as relighting is still not possible in VDTM framework. This is due to all the lighting effect is baked into the texture. We present a novel approach for estimating the illumination and reflectance properties of virtual objects captured using consumer‐grade RGB-D cameras. This method is implemented within a fully automatic content creation pipeline that can generate photorealistic objects suitable for integration with virtual reality scenes with dynamic lighting. The geometry of the target object is first reconstructed from depth images captured using a handheld camera. To get nearly drift free texture maps of the virtual object, a set of selected images from the original color stream is used for camera pose optimization. Our approach further separates these images into diffuse (i.e, view‐independent) component and specular (i.e., view‐dependent) components using low‐rank decomposition. The lighting conditions during capture and reflectance properties of the virtual object are subsequently estimated from the computed specular maps. By combining these parameters with the diffuse texture, the reconstructed object can then be rendered in a real‐time scene that plausibly replicates the real world illumination or with arbitrary lighting that varies in direction, intensity, and color