Adaptation of deep neural network acoustic models in automatic speech recognition

Ph.D.In this thesis, DNN acoustic model adaptation is investigated. Performance of automatic speech recognition (ASR) systems may be affected by mismatch between the systems and target speech. This mismatch is caused by acoustic variability in the speech under different input conditions, e.g., speaker, environment, speaking style, and emotional state. Techniques of acoustic model adaptation play a key role in reducing the mismatch. Currently, most of the widely used adaptation methods are proposed for speaker variability. These speaker adaptation techniques can also be applied to deal with acoustic variability contributed by other identifiable input conditions. However, acoustic variability may be joint effect of latent input conditions, which are often unidentifiable or even unknown. This kind of variability is referred to by “unknown variability” in this thesis. It is difficult to deal with by speaker adaptation methods.This research is motivated to address two main issues in acoustic model adaptation. The first issue is the efficiency of parameter estimation for model-based speaker adaptation. For example, the commonly used learning hidden unit contributions (LHUC) based DNN adaptation requires multiple decoding passes and explicit parameter estimation during test-time adaptation process. Efficient adaptation could be achieved either by reducing the amount of required adaptation data or by developing fast algorithms of parameter estimation. Bayesian learning for parameter estimation is applied to improve robustness for LHUC adaptation. In this way, relatively small amount of adaptation data can be used for higher efficiency. The proposed Bayesian LHUC (BLHUC) adaptation makes use of variational approximation of posterior distribution to explicitly model the parameter uncertainty. Moreover, Gaussian mixture model (GMM) based distribution can be used to extend the BLHUC adaptation. On the other hand, an efficient adaptation approach is proposed to directly predict the LHUC parameters on the fly. The clumsy adaptation process is no longer required in the test stage. Experiments conducted on 300-hour Switchboard corpus demonstrate the advantages of the proposed BLHUC adaptation and LHUC feature prediction over the standard LHUC adaptation.The second issue is the effective realization of adaptation to deal with unknown variability. Latent Dirichlet allocation (LDA) based clustering can be used to address this issue. Recurrent neural network latent Dirichlet allocation (RNNLDA) is proposed to relax the limitation of applying conventional LDA to speech data. Alternatively, unknown variability can be directly modeled by auto-encoder based variability feature extractor (AEVFE) without applying clustering. Experiments conducted on DARPA Gale Mandarin Chinese broadcast speech, Aurora 4, and UASpeech databases show that the RNN-LDA clustering and the AEVFE bottleneck features are able to effectively encode the information of unknown variability. Moreover, each of them can be jointly used with LHUC adaptation in DNN acoustic model for further performance improvement.本文研究課題是DNN聲學模型的自適應。自動語音識別系統與目標語音之間的不匹配或會影響系統的性能。這種不匹配是由不同輸入條件下的語音中的聲學變異導致，例如說話人，環境，說話方式，以及情緒狀態。聲學模型自適應技術對降低這種不匹配起到關鍵作用。當下大部分廣泛應用的自適應方法都用於處理說話人變異。這些說話人自適應技術也可用於處理由可辨認的輸入條件導致的聲學變異。然而，聲學變異有時是潛在的輸入條件的共同作用。這通常是不可辨認甚至未知的。本文用“未知變異”指代該種變異。這是難以由說話人自適應方法處理的。本研究旨在處理聲學模型自適應的兩個主要問題。第一個問題是基於模型的說話人自適應參數估計的效率。例如常用的基於隱單元貢獻學習（LHUC）的DNN自適應在測試過程要求多次的解碼和顯式的參數估計。高效的自適應能夠通過兩種方式獲得：減少要求的自適應數據量；開發參數估計的快速算法。參數估計的Bayesian學習被用於提高LHUC自適應的健壯性。通過這種方法，相對少的數據能用於獲得更高效率。所提出的Bayesian LHUC（BLHUC）自適應利用後驗分佈的變分近似來顯式地模擬參數的不確定性。基於高斯混合模型（GMM）的分佈也能用於擴展BLHUC自適應。另一方面，本文也提出了一種直接動態地預測LHUC參數的快速的自適應方法。這使得測試階段不再需要笨重的自適應過程。在300小時Switchboard數據庫上的實驗顯示出了所提出的BLHUC自適應以及LHUC特徵預測相較於標準LHUC自適應的優點。第二個問題是有效地實現對未知變異的自適應。基於隱Dirichlet分佈（LDA）的聚類能用於解決該問題。本文提出遞歸神經網絡-隱Dirichlet分佈（RNN-LDA）用於緩和在語音數據上應用傳統的LDA時的限制。另一種選擇是通過基於自編碼器的變異特徵提取器（AEVFE）直接地對未知變異進行建模，而無需使用聚類方法。在DARPA Gale國語廣播語音，Aurora 4，以及UASpeech數據庫上的實驗表明，RNN-LDA聚類以及AEVFE瓶頸特徵皆可有效地對未知變異信息進行編碼。而且他們能夠跟LHUC自適應共同用於DNN聲學建模來獲得進一步的性能提升。Xie, Xurong.Thesis Ph.D. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 93-108).Abstracts also in Chinese.Title from PDF title page (viewed on November 2, 2021)