Tele-immersive interaction with intelligent virtual agents based on realtime 3D modeling

Abstract—To enable intelligent agents interacting smoothly with human users, researchers have been deploying novel interaction modalities (e.g. non-verbal cue, vision and touch) in addition to agents ’ conversational skills. Models of multi-modality interaction can enhance agents ’ real-time perception, cognition and reaction towards the user. In this paper we report a novel tele-immersive interaction system developed using real-time 3D modelling techniques. In such system user’s full body is reconstructed using multi-view cameras and CUDA based visual hull reconstruction algorithm. User’s mesh model is then loaded into a virtual environment for interacting with an autonomous agent. Technical details and initial results of the system are illustrated in this paper. Following that a novel interaction scenario is proposed which links the virtual agent with a remote physical robot who takes the role of mediating interactions between two geographically separated users. Finally we discuss in depth the implications of such human-agent interaction and possible future improvements and directions. Index Terms—intelligent virtual agent, tele-immersion, interaction, real-time modeling I