Disordered Speech Recognition and the use of Multi-modal Information

Despite the rapid development of automatic speech recognition (ASR) technologies in the past few decades, recognition of disordered speech is a highly challenging task. Speech disorders such as dysarthria affect millions of people worldwide and the quality of their life. Such people often experience physical disabilities and other co-occurring medical conditions at the mean time. Their difficulty in using keyboard, mouse and touch screen based user interfaces makes speech controlled assistive technologies more natural alternatives.Disordered speech presents a wide spectrum of challenges to current deep neural networks (DNNs) based speech recognition technologies that predominantly target normal speech. First, a large mismatch between disordered and normal speech is often observed. People suffering from speech impairments tend to use shorter utterances based on isolated words and simple commands, due to the fatigue they encounter when speaking. This limits the long range temporal contexts that current DNN based ASR systems designed for normal speech can exploit. Second, the underlying neuro-motor conditions, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of disordered speech required for ASR system development. Finally, the large variation among speakers with diverse impairment characteristics, severity levels and in different stages of speech disorder progression creates large variation in disordered speech data.In this thesis, we attempt to address the above issues as described below. First, motivated by the large mismatch between normal and disordered speech, a systematic investigation of neural network architecture designs targeting dysarthric speech recognition is conducted. Automatic neural architecture search (NAS) techniques are further used to refine the structural configurations. Second, in order to address the data sparsity problem in disordered speech recognition system development, and inspired by the success of data augmentation techniques widely reported in normal speech recognition tasks, data augmentation techniques designed to model the spectral-temporal level deviation of disordered speech from normal speech are used. Third, in order to model the large variability among disordered speakers in both the original and augmented data, model based DNN adaptation methods represented by, for example, learning hidden unit contributions (LHUC) based speaker adaptive training (SAT) are further applied. Bayesian speaker adaptation approaches were also employed to facilitate both rapid, instantaneous adaptation to individual speakers' voice characteristics.Human speech generation is inherently a bimodal process based on audio-visual representation. The visual information is invariant to acoustic signal corruption and can provide complementary information to speech recognition. This motivates our use of visual information to improve speech recognition performance for disordered speech by developing audio-visual speech recognition (AVSR) systems. The movements of articulators with their underlying articulatory information, which is highly related with speech production, is another useful resource of features for assisting speech recognition system development. Thus the articulatory information is explored as well to guide the disordered speech recognition systems development.In this thesis, a novel Bayesian gated neural network (BGNN) based AVSR architecture is proposed to robustly integrate the visual information while handling the data sparsity when there is very limited audio-visual training data. A cross-domain visual feature generation approach is proposed to produce visual features for those speakers who do not have video data, together with data augmentation techniques to enlarge the audio-visual training data for AVSR system development. At last, this thesis explores the use of articulatory features for disordered speech recognition. Experiment results show that by using multi-modal sources such as visual and articulatory information can improve the recognition performance of disordered speech.儘管在過去數十年間自動語音識別技術快速發展，障礙語音識別仍然是一個非常具有挑戰性的工作。言語障礙中，比如構音障礙，影響了全世界千千萬萬的人以及他們的生活質量。具有這种病患的人通常會有身體殘疾或者同時伴有其他併發症。他們很難使用鍵盤、鼠標或者帶有觸摸屏幕的裝置，這使得語音控制輔助技術成為一個自然選擇。障礙語音對當前主要針對正常語音的基於深度神經網絡的語音識別技術提出了廣泛的挑戰。首先，經常觀察到障礙語音和正常語音之間存在很大的不匹配。患有語言障礙的人傾向於使用基於孤立單詞和簡單命令的較短話語，因為他們在說話時會感到疲勞。這限制了當前為正常語音設計的基於深度神經網絡的語音識別系統利用長時間上下文的能力。其次，潛在的神經運動狀況，通常會伴隨身體殘疾而發生，導致難以收集大量障礙語音來開發語音識別系統。最後，具有不同損傷特徵、嚴重程度和語音障礙進展的不同階段的說話者之間的巨大差異會導致障礙語音數據的巨大差異。在本論文中，我們嘗試解決上述問題。首先，正常語音和障礙語音之間存在巨大不匹配，我們系統檢驗了針對構音障礙語音識別的神經網絡結構。自動神經網絡結構搜索技術用來進一步細化結構配置。其次，為了解決障礙語音識別系統開發中的數據稀疏問題，並受到在正常語音識別任務中廣泛報導的數據增強技術成功的啟發，我們對正常語音採用了數據增強技術，來模擬正常語音與障礙語音的頻譜-時間偏差。第三，為了對原始數據和增強數據中障礙語音說話者之間的巨大差異性進行建模，進一步應用了基於模型的深度神經網絡自適應方法，例如基於學習隱藏單元貢獻的說話者自適應訓練。貝葉斯說話人自適應方法也被用來針對個別說話人的語音特徵進行快速、即時的自適應。人類的發聲內生性的就是一種基於語音-視覺雙模態信息表示的過程。視覺信息相對於語音信號損壞具有不變性，並且可以給語音識別提供補充信息。這激勵我們使用視覺信息來開發語音-視覺語音識別系統來改善障礙語音的語音識別性能。發音器官的運動及其潛在發音信息與語音產生高度相關，是輔助語音識別系統開發的另一個有用的特徵資源。因此，發音信息也被探索以指導障礙語音識別系統的開發。這篇論文提出了一個新穎的基於貝葉斯門控神經網絡的語音-視覺語音識別系統框架，以此來同時解決兩個問題，一是如何魯棒地利用視覺信息，二是如何利用非常有限的語音-視覺數據來搭建障礙語音的語音-視覺語音識別系統。這篇論文還提出了一種跨域視覺信息生成方法，來為那些沒有視頻數據的說話人生成視覺特征，同時還利用了數據增強技術來擴充語音-視覺訓練數據，用來搭建語音-視覺語音識別系統。最後，這篇論文探索了如何使用發音特征來輔助障礙語音識別。實驗表明，多個模態的數據源比如視覺或者發音信息可以提高障礙語音的識別效果。LIU, Shansong.Ph.D. Chinese University of Hong Kong 2021.Includes bibliographical references (leaves )Abstracts in English and Chinese.Title from PDF title page (viewed on ...