Network Support for Social 3-D Immersive Tele-Presence with Highly Realistic Natural and Synthetic Avatar Users

The next generation in 3D tele-presence is based on modular systems that combine live captured object based 3D video and synthetically authored 3D graphics content. This paper presents the design, implementation and evaluation of a network solution for multi-party real-time communication of these types of content. This prototype includes a UDP/TCP multi-streaming kernel that includes media synchronization support, packet scheduling, loss resilient real-time transmission and an easy to use blocking and non-blocking API. To compress the live reconstructed 3D data streams that represent the natural user, two categories of 3D mesh codecs were integrated: a highly adaptive real-time geometry driven mesh codec and a fast single rate codec that provides better performance at high resolutions. Subjective tests with 16 subjects indicate that only modest perceptual degradation of the highly realistic 3D natural user is introduced, especially when the users in the virtual world are at a distance. We developed a session management protocol for setting up streams based on the specific 3DTI capabilities allowing device scalability from light (render only) to heavy clients (rendering and 3D Capturing). Additionally, a distributed messaging system via web-sockets and cloud infrastructures based on publish and subscribe was integrated for real-time delivery of avatar and other AI data