Automatic assessment of speech and language impairment in spontaneous narrative speech

Ph.D.People who suffer from speech and language impairment face serious problems in their daily life and work. Currently, subjective auditory analysis of speech is one of the most commonly used methods for the assessment of relevant disorders and diseases. This is a challenging task because it requires not only clinical knowledge but also appropriate linguistic and cultural background. Speech technology can be used to enable objective assessment of natural speech and improve the efficiency and effectiveness of clinical diagnosis and rehabilitation.The research presented in this thesis is focused on fully automatic assessment of speech and language impairment using speech processing and machine learning techniques. Speech data involved in the research are from people with aphasia (PWA). They reveal a variety of symptoms of speech and language impairment. We focus on Cantonese, which is a representative and influential Chinese dialect. There have been extensive work on automatic assessment on aphasic speech of western languages, while studies on Cantonese aphasic speech remain under-investigated.The thesis is organized into three parts. The first part targets at speech assessment with explicit text and acoustic features. To begin with, an in-domain automatic speech recognition (ASR) system is developed. Multi-task learning strategy is applied to improve the performance of aphasic speech recognition. It alleviates the problem of lacking proper domain-matched training data. The ASR system lays the foundation for subsequent extraction of reliable text features and acoustic features. Word embedding and siamese networks are applied to extract robust text features for characterizing language impairment. The text features are derived from the ASR decoding result on impaired speech, aiming to quantify the content discrepancy between narrative speech from impaired speakers and those from unimpaired speakers. In order to mitigate the effect of ASR errors, we propose to incorporate confusion networks of ASR output into the computation of text features. On the other hand, supra-segmental duration features computed from ASR time-alignment as well as a set of conventional acoustic features are used to characterize the atypical prosody and voice quality. With a combination set of text features and acoustic features, automatic prediction of subjective assessment scores of PWA is achieved. The perdition results show high correlation with ground-truth assessment scores.The second part of work investigates the use of phone posteriorgrams from ASR in classifying the severity level of aphasia. This method aims at speech and language impairments that can not be explicitly detected from erroneous ASR output, known as paraphasias. Posteriorgrams are found to be able to capture paraphasia, rhythm of speech and voice abnormality in impaired speech. A siamese network and a 2-channel convolutional neural network (CNN) are applied to learn impairment-related features from a pair of weakly and strongly constrained posteriorgrams.In the final part of this thesis, an attempt to end-to-end speech assessment is made. Conventional assessment method relies on independent procedures of feature design, feature selection and classification/regression. Speech and language impairments may not be exhaustively covered by hand-crafted features. To overcome this shortcoming, deep neural networks (DNNs) are applied to realize an end-to-end mapping from fundamental acoustic features or text sequences of impaired speech to the assessment result. Impairment-related features are learned implicitly by the neural network model. Experimental results show that the end-to-end approach could achieve comparable performance to the conventional assessment approach. The class activation mapping (CAM) visualization also confirms that the acoustic features learned with CNN are similar to hand-crafted features.言語和語言障礙患者的日常生活和工作會受到嚴重影響。目前，對語音進行主觀聽覺分析是評估相關障礙和疾病最常用的手段之一。但實現這種評估方式對評估者有一定的要求。他們不僅需要有關疾病的臨床知識，而且還需要對相關的語言和文化背景有較好的理解。語音技術的發展為建立基於自然語音的客觀評估系統提供了可能性。該系統以提高臨床診斷和康復過程的效率為目標。本研究致力於使用先進的語音處理和機器學習技術對言語和語言障礙進行全自動的評估。在研究中使用的語音數據全部來自於講廣東話的失語症患者。他們的語音中包含多樣化的言語和語音障礙症狀。現有研究多集中於對西方國家失語症患者的語音進行自動評估，而針對講廣東話的失語症患者的相關研究卻很少見。本論文的研究包括三個部分。第一部分旨在使用可被顯性提取的文字和聲學特徵進行評估。首先，我們搭建了一個領域匹配的語音識別系統。該系統使用多任務學習的方法來提高對失語症語音的識別準確率。這種方法有助於緩解缺乏適當的領域匹配數據訓練聲學模型的問題。該語音識別系統為後續提取用於評估的可靠的文本和聲學特徵奠定了基礎。為了對語言障礙進行刻畫，我們使用詞嵌入技術和孿生神經網絡來提取文本特徵。這些特徵可以從失語症語音的識別結果中得出，可以衡量失語症病人講述故事與正常人講述故事在內容上的差異。為了減輕語音識別錯誤對文本特徵提取的影響，我們將語音識別中的混淆網絡納入文本特徵的計算中。另一方面，從語音識別的時間對齊中提取的超音段時間特徵和一組基本聲學特徵分別被用來刻畫失語症語音中韻律和嗓音的異常。通過結合之前提取的文本特徵，我們可以實現對失語症患者的主觀評估分數進行自動預測。實驗結果表明自動評估得到的分數與真實主觀評估的分數高度相關。第二部分工作研究了提取自語音識別系統的後驗概率圖在失語症嚴重程度的分類任務中的有效性。這個方法主要針對那些在語音識別的結果中不易被顯式檢測出的言語和語言障礙特徵，即誤語症現像。後驗概率圖特徵可以反映失語症語音中的誤語症現像及語速和嗓音的異常。我們利用孿生神經網絡和雙通道的卷積神經網絡來學習強-弱後驗概率圖對中與評估目標直接相關的特徵。在本論文的最後一部分中，我們嘗試了端到端的語音評估方法。傳統的評估方法依賴於獨立的特徵設計、特徵選取及分類/回歸過程。人為設計的特徵難以涵蓋全部的言語和語言障礙現像。為了克服這一缺點，我們利用深度神經網路來實現從失語症語音的基本聲學特徵或文本數據到評估結果的端到端映射。神經網絡模型能夠隱式地學習與病理相關的特徵。實驗結果表明，端到端的方法可以達到與傳統評估方法可比的性能。同時，基於類激活映射的可視化方法也證實了通過卷積神經網絡所學到的聲學特徵與人為設計的特徵相近。Qin, Ying.Thesis Ph.D. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 133-156).Abstracts also in Chinese; appendix A includes Chinese.Title from PDF title page (viewed on 26, October 2021)