Automatic Speech Recognition System Using Acoustic and Visual Signals

Automatic speechreading systems use both acoustic and visual signals to perform speech recognition. In previous work, we have shown how visual speech can improve recognition accuracy of automatic speech recognition [10] and have described an algorithm based on deformable templates that accurately infers lip dynamics [1]. In this paper we present a complete speechreading system, which is able to record an utterance using a standard color video camera, preprocess both the audio and video signal, and perform speech recognition. This system is based on new algorithms for finding the talker's face and mouth and an improved template algorithm for tracking the lips. We will also compare the results from our new system with our previous work and discuss various strategies for integration of the two modalities