A Probabilistic Model for Generating Realistic Lip Movements from Speech

Abstract The present work aims to model the correspondence between facial motion and speech. The face and sound are modelled separately, with phonemes being the link between both. We propose a sequential model and evaluate its suitability for the generation of the facial animation from a sequence of phonemes, which we obtain from speech. We evaluate the results both by computing the error between generated sequences and real video, as well as with a rigorous double-blind test with human subjects. Experiments show that our model compares favourably to other existing methods and that the sequences generated are comparable to real video sequences. 1 Introduction Generative systems that model the relationship between face and speech offer a wide range of exciting prospects. Models combining speech and face information have been shown to improve automatic speech recognition [4]. Conversely, generating video-realistic animated faces from speech has immediate applications to the games and movie industries. There is a strong correlation between lip movements and speech [7,10], and there have been multiple attempts at generating an animated face to match some given speech realistically [2,3,9,13]. Studies have indicated that speech might be informative not only of lip movement but also of movement in the upper regions of the face [3]. Incorporating speech therefore seems crucial to the generation of true-to-life animated faces