Posteriorgram-to-acoustic modeling for unconstrained voice conversion with deep learning

Ph.D.This thesis mainly investigates the use of posteriorgram-to-acoustic modeling forunconstrained Voice Conversion (VC) with deep learning. Many existing systems arelimited by some constraints including requirement of parallel data, requirement oflarge amounts of language-dependent training data, which also affects speech qualityand speaker similarity. Therefore, this thesis concentrates on two goals that alleviatethe said constraints: 1) improving speech naturalness and speaker similarity of theconverted speech; 2) improving practicality and flexibility of VC systems.For VC with parallel data, many previous efforts (e.g. GMMs, DNNs) model therelationship between the paired non-target speech and target speech at the frame-basedlevel directly, but context dependency among the neighboring frames is ignored. To addressthis issue, two proposed approaches are: 1) an augmented DNN-based approachwith dynamic features; 2) a sequence-based method using Deep Bidirectional LongShort-Term Memory based recurrent neural networks (DBLSTMs) to model not onlythe frame-wise relationship, but also long-range context dependency. Experimentsshow that the DNN-based approach with dynamic features (DNN-DYN) can improvethe naturalness of the converted speech, increasing the Mean Opinion Score (MOS)from 2.3 to 2.7. Furthermore, the DBLSTM-based approach can further improve the MOS from 2.7 to 3.2 and speaker similarity (ABX percentage of DNN-DYN and DBLSTM: 32.0% vs. 53.0%).To build an unconstrained VC without parallel data, a framework with Posteriorgramto-Acoustic Modeling (PAM) is proposed. The novelty is to bridge the non-target and target speakers by means of Phonetic PosteriorGrams (PPGs), which provide a good representation for speaker-independent information. The framework has two main advantages: 1) freeing up the constraint of parallel training data, thus offering practicality; 2) freeing up the constraint of any new model training for different nontarget speakers mapping to a fixed target speaker, thus offering flexibility. Experiments show that the PAM-based approach performs better than state-of-the-art DBLSTMbased approach with parallel data system in both speech quality (MOS of DBLSTM and PAM: 3.1 vs. 3.8) and speaker similarity (ABX percentage of DBLSTM and PAM: 17.8% vs. 52.0%).We push further to obtain an unconstrained VC across languages. PAM is well suited for cross-lingual tasks, because it is assumed that they can represent small speech segments common across languages. Experiments suggest that the PAM-based approach on cross-lingual conversion can achieve very similar performance to that on intra-lingual conversion.本文主要研究了后驗概率到聲學參數模型（Posteriorgram-to-Acoustic Modeling,PAM）在基於深度學習語音轉換中的應用。許多已有的系統有一些限制條件，包括需要平行語句，大量訓練數據，依賴於語言種類等。這些限制會影響到語音質量和說話人相似度。因此，本文主要著重於兩個方面：1）提高被轉換語音的自然度和說話人相似度；2）提高系統的實用性和靈活性。對於基於平行語句的語音轉換，已有的很多方法（比如高斯混合模型，深度神經網絡）直接在非目標語音和目標語音的幀層面進行建模，但忽略了語音幀與幀之間的上下文關係。為了解決這個問題，我們提出了兩個方法：1）用動態特征強化深度神經網絡（DNN-DYN）；2）用基於序列的深度雙向長短時記憶神經網絡（DBLSTM）對長範圍的上下文依賴關係進行建模。實驗表明使用動態特征的深度神經網絡方法可以提高被轉換語音的自然度，將主觀分數（MOS）從2.3提高到2.7。基於DBLSTM的方法可進一步把主觀分數從2.7提高到3.2。並且在說話人相似度ABX實驗中，DBLSTM方法好於DNN-DYN方法（DNN-DYN和DBLSTM的ABX比重：32.0% vs. 53.0%）。為了建立不受平行語句限制的語音轉換系統，我們提出了一個基於后驗概率到聲學參數模型的框架。新穎性在於利用后驗概率（Phonetic Posteriorgrams）來連接非目標說話人和目標說話人。后驗概率可以很好的表示語音中與說話人無關的信息。這個框架有兩個主要優點：1）除去平行語句的限制，因此提高實用性；2）可用於任何非目標說話人對特定目標說話人的語音轉換，因此提高靈活性。實驗表明PAM方法優于基於平行語句的DBLSTM方法。在自然度方面，DBLSTM和PAM主觀分數對比：3.1 vs.3.8。在說話人相似度方面，DBLSTM和PAM 的ABX比重：17.8% vs. 52.0%。進一步，我們建立不受語言限制的語音轉換系統。由於小的語音片段可在語言中共享，因此PAM可以很好的適用於跨語言任務。實驗表明我們的PAM方法在跨語言任中的表現比得上在非跨語言任務中的表現。Sun, Lifa.Thesis Ph.D. Chinese University of Hong Kong 2017.Includes bibliographical references (leaves 93-107).Abstracts also in Chinese.Title from PDF title page (viewed on …)