AudioVisual Speech Recognition using Red Exclusion and Neural Networks

Automatic speech recognition (ASR) performs well under restricted conditions, but performance degrades in noisy environments. Audio-Visual Speech Recognition (AVSR) combats this by incorporating a visual signal into the recognition. This paper briefly reviews the contribution of psycholinguistics to this endeavour and the recent advances in machine AVSR. An important first step in AVSR is that of feature extraction from the mouth region and a promising new technique is presented. This paper examines how useful this extraction technique in combination with several integration architectures and compares it with competing techniques, demonstrates that vision does in fact assist speech recognition when used in a linguistically guided fashion, and gives insight into remaining issues