Modeling Efficient Deep Representations for Human Digitization with Affordable Setup

A Digital human is a photorealistic 3D human model. Digital humans have a wide range of potential applications. For example, they can be used in free-viewpoint videos that allow users to freely choose the viewpoint position and orientation. Additionally, digital humans can be used in telepresence, where people in different locations can feel as though they are in the same space. Furthermore, we can enable more immersive virtual meetings than traditional videoconferencing software such as Zoom. In current systems, to create a digital human, we need to first capture the subject using a dense array of cameras, and apply multi-view stereo algorithm to obtain the 3D scan. Next, a 3D artist should clean up the 3D scan, rig, and animate it. Lastly, lighting and rendering are performed to create the video of a digital human. Those procedures require expensive hardware setups. Also, they are very complicated, time-consuming, and need to be performed in very specialized and constrained environments. This means that the current system of creating digital humans is not affordable to everyone. Enabling affordable human digitization can facilitate broader access to immersive digital experiences. Therefore, this thesis aims to model an efficient representation that can reconstruct a 3D human with an affordable setup. To this end, this thesis makes efforts from two different and important aspects: human performance capture and avatar synthesis. Human performance capture is the process of capturing and reproducing the movements, postures, appearances, and facial expressions of human performers in digital form. Avatar synthesis is the process of synthesizing the human subject in new poses. For human performance capture, this thesis mainly works on achieving identity generalization from sparse camera inputs.The goal here is to create a representation that can synthesize the novel view of an unseen human subject using just a few sparse multi-view inputs. This direction is especially important to realize the immersive AR and VR experience where we need to instantly capture and reconstruct the changing appearance of the subject. This thesis first explores volumetric representation to achieve generalization. Here, we enhance image synthesis quality as well as temporal coherency by leveraging complementary information between different time frames. Second, this thesis presents neural radiance fields conditioned on a parametric human template. Human performance capture of an arbitrary human at inference time is highly challenging due to the heavy occlusions and dynamic articulations of the body parts. By leveraging the human prior, we achieve robust performance. We further improved the rendering quality as well as the cross-dataset generalizability by incorporating image-based rendering. For affordable avatar synthesis, this thesis presents an animatable representation that can be learned solely from the RGB video sequences. Specifically, this work proposes a two-surface parameterized light field that can synthesize the subject in new poses with high quality in real time. Finally, this thesis presents the practical application of our 3D human digitization methods in the domain of telemedicine. To demonstrate this, the proposed human performance capture methods are applied to a dataset that records a patient with Parkinson's disease. The results illustrate that our method can synthesize plausible novel views using sparse camera inputs. This highlights the potential of our human digitization methods to further expand the field of telemedicine.Doctor of Philosoph