A Flexible Neural Encoder-Decoder Framework for Multi-faceted Voice Transformation

Voice Transformation (VT) aims at modifying some components of a voice signal while retaining other components. This thesis proposes the use of a novel, neural encoder-decoder (NED) framework for VT, which offers maximum advantage of preserving invariant content information, while a bank of encoders is used to extract modifiable facets of a voice signal. The decoder aggregates both invariant and modified components of speech to regenerate the signal and in doing so accomplishes the objective of VT. This framework is shown to be the first unified framework able to conduct various types of VT tasks and differs from previous approaches in offering the following advantages: (1) it is highly flexible and extensible in system structure; (2) it is data efficient. To demonstrate the effectiveness and versatility of the proposed framework, and its effectiveness and efficiency in VT, we conduct our investigation through three different tasks of accent deduction (AD), speaker identity conversion (SIC) and singing voice conversion (SVC).The first task is AD, which aims at transforming non-native utterances to sound as if the speaker had a native accent. We use NED to disentangle accent and voice identity, and carefully design the training schemes to make full use of out-domain data and the in-domain data and propose a reference-free AD system which does not need a native reference utterance during conversion. Experiments show that the proposed approach effectively converts Hindi-accented English into native American English.The second task is SIC, which aims at modifying a speech signal to sound as if it was uttered by some target speaker, while keeping the linguistic contents unchanged. We focus on tackling the challenging and promising any-to-many/any SIC problem using one model. Under the NED framework, we propose an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel SIC approach. We cascade a bottle-neck feature extractor (BNE) with a multi-speaker sequence-to-sequence synthesis module, where the former computes speaker-agnostic content features from a speech signal and the latter aggregates content information and speaker representation to generate speech. To accomplish any-to-any conversion, we pre-train an encoder with a speaker recognition criterion and use it to generate speaker representations for unseen speakers.The third task is SVC, a technique aiming at converting the voice of one singer to that of other singers without changing the underlying content and melody. Since the supra-segmental characteristics (e.g., duration and melody) should be retained during conversion, an encoder which can obtain frame-level content features from a source signal is used as a content encoder. Since SVC requires the prosodic information to be unchanged during conversion, we use two additional encoders to extract melody and loudness information. A novel double U-shape network structure is applied in the decoder to fully fuse melody and loudness information into the decoding process.The experimental results from the three studied tasks validate that the NED framework is versatile, effective and efficient for VT problems. Hence, this framework can also be adopted for other VT tasks such as emotional voice conversion, whisper-to-speech transformation, speech-to-singing conversion, cross-lingual voice cloning, etc.語音變換(VT)旨在修改語音信號的某些成分，同時保留其他成分。本論文提出使用一種新穎的神經編碼器-解碼器(NED)框架進行VT，它提供了保留不變內容信息的最大優勢，同時使用一組編碼器來提取語音信號的可修方面。解碼器聚合語音的不變和修改分量以重新生成信號，從而實現VT的目標。該框架被證明是第一個能夠執行各種類型的VT任務的統一框架，與以前的方法不同，它具有以下優點：（1）系統結構高度靈活和可擴展；(2)數據高效；為了證明所提出框架的有效性和多功能性，以及它在VT中的有效性和效率，我們通過三個不同的任務進行調查：口音扣減(AD)、說話人身份轉換(SIC)和歌聲轉換(SVC)。第一個任務是AD，其目的是將非母語話語轉換為說話者俱有母語口音的聲音。我們使用NED來解開口音和語音身份，並精心設計訓練方案以充分利用域外數據和域內數據，並提出一個在轉換過程中不需要本地參考話語的無參考AD系統。實驗表明，所提出的方法有效地將印地語口音的英語轉換為美國本土英語。第二個任務是SIC，它旨在將語音信號修改為聽起來好像是某個目標說話者所說的那樣，同時保持語言內容不變。我們專注於使用一個模型解決具有挑戰性和前景的任意對多/任意SIC問題。在NED框架下，我們提出了一種任意對多位置相關、序列到序列(seq2seq)、非並行SIC方法。我們將瓶頸特徵提取器(BNE)與多說話人序列到序列合成模塊級聯起來，其中前者從語音信號中計算與說話人無關的內容特徵，後者聚合內容信息和說話人表示以生成語音。為了完成任意到任意轉換，我們使用說話人識別標準預訓練編碼器，並使用它為看不見的說話人生成說話人表示。第三個任務是SVC，一種旨在在不改變底層內容和旋律的情況下將一個歌手的聲音轉換為其他歌手的聲音的技術。由於在轉換過程中應保留超分段特徵（例如，持續時間和旋律），因此可以從源信號中獲取幀級內容特徵的編碼器用作內容編碼器。由於SVC要求在轉換過程中韻律信息保持不變，我們使用兩個額外的編碼器來提取旋律和響度信息。解碼器採用了一種新穎的雙U形網絡結構，將旋律和響度信息充分融合到解碼過程中。三個研究任務的實驗結果驗證了NED框架對於VT問題是通用的、有效的和高效的。因此，該框架也可以用於其他VT任務，例如情感語音轉換、耳語到語音的轉換、語音到唱歌的轉換、跨語言語音克隆等。LIU, Songxiang.Ph.D. Chinese University of Hong Kong 2021.Includes bibliographical references (leaves )Abstracts in English and Chinese.Title from PDF title page (viewed on ...