Projective multi-texturing for integrated real-time 3D reconstruction and rendering of a person

The ability to capture and reconstruct a person in real-time 3D environments, offers significant improvements to fidelity and empathetic engagement in real-time visual communication/collaboration systems, provided it can be achieved at a fast enough rate to deliver the natural animation of the captured person. To date the majority of approaches have sought to adapt techniques from film and TV special effects, which generate either volumetric models, which may, or may not, be tessellated and textured before rendering, or polyhedral (space carved), viewpoint independent textured 3D geometric models. These approaches are refined, through optimization and migration to distributed architectures, and frequently utilize GPU processing to further enhance key sub-processes to achieve (near) real-time performance. A significantly smaller number of approaches have applied image based rendering techniques to generate viewpoint dependent reconstructions within the rendering pipeline. On current hardware, such techniques offer reasonable performance and acceptable quality. We view the process of surface forming and tessellation as computationally expensive and unnecessary, arguing that such approaches fail to exploit the future potential of GPU based processing systems. Conversely, image based rendering approaches often utilise graphics hardware effectively, but lack the quality and flexibility of viewpoint independent methods. Therefore, we present a novel technique which combines the principles of projected image volumetric reconstruction, without surface forming, within an image based rendering approach, thereby integrating reconstruction and rendering entirely within the graphics pipeline to deliver fast, good quality 3D reconstruction. This approach aims match or better the performance of current approaches by effectively exploiting the GPU, and is potentially scalable with the relentless increase in GPU processing delivered through ever increasing parallelism. We describe the technique, both theoretically and in implementation, and present quantitative assessments of performance, and potential for scalability. We conclude with a discussion of the future potential of the approach and critical evaluation of outstanding issues