Direct Face-to-Face Communication Between Real and Virtual Humans

Interaction in graphical systems is becoming more and more multimodal. Use of the conventional mode of 2D widget-mouse or keyboard interaction is not very natural for many applications. For the comfort of disabled persons who may not be able to use the hand input devices, it is necessary to explore means for interactive controls for them. In this paper, we discuss real time interaction through facial input and face-to-face communication in a virtual dialog. The paper presents our method of extracting dynamic facial movements which can be hooked as controls for a desired application. For example, in performance driven facial animation, the method enables recognition of facial expressions of a real person which are appropriately mapped as controlling parameters to simulate facial expressions of a synthetic actor in real time. In other applications, the extracted parameters can provide real time estimates of positions and orientations in a virtual scene. The method requires a video camera input and extracts motion parameters through a small set of visually tracked feature points. In the face-to-face virtual dialog, recognition acts like a module for acquiring emotional states of a real actor. This can be integrated with other modules for the interpretation of the response with voice and emotions of the virtual actor. This gives potential for an interface system with human like agent interacting with user in spoken language with facial expressions