Lip tracking using pattern matching snakes

Lip reading is an important tool used by hearing im-paired people to increase their understanding of spoken lan-guage. Linguistics experiments show that the features of speech that are degraded by noise in audio systems are the features of speech that are most distinct visually. An auto-matic visual speech recognition system should therefore be a beneficial complement to an audio speech recognition sys-tem when the audio system is used in a noisy environment. An automatic lipreading system requires a robust method to track lips from the image sequence, regardless of the vari-ations of lip shapes, colours and movement amongst speak-ers. One of the most common lip tracking methods is the use of active contour models or snakes to detect outer lip con-tours. This paper proposes a robust and adaptable lip track-ing method that uses a combination of snakes and a 2D tem-plate matching technique. The snake, an energy minimising spline, is driven by 2D template matching techniques to find the expected lip contour of a specific speaker. Our experi-ments show that the technique can track the unadorned lips in various colours and shapes of speakers, including the lips of a bearded speaker. 1