Deep Generative Models for Natural Language Generation

Thesis (Ph.D.)--University of Washington, 2020Natural language generation plays an important role in language intelligence, which is an essential topic of artificial intelligence over the past years. Recent advances in generative models combining with deep neural networks have achieved tremendous successes in many natural language generation tasks. Establishing suitable and effective generative models is the key challenge for researchers to fulfill different language generation purposes under varied application scenarios. This thesis focuses on investigating and providing better deep generative models with respect to various natural language generation tasks. This thesis consists of two parts. The first part explores the ranking-based generative adversarial network for generating texts. We first examine limitations of the commonly used Generative Adversarial Networks (GANs) on text generation tasks, and propose a novel ranking-based generative adversarial network, RankGAN, for generating high-quality language descriptions. Rather than training the discriminator to learn and assign an absolute binary predicate for an individual data sample, the proposed RankGAN is able to analyze and rank a collection of human-written and machine-written sentences by giving a reference group. Concretely, by viewing a set of data samples collectively and evaluating their quality through relative ranking scores, the discriminator is able to make a better assessment which in turn helps to learn a better generator for text generation tasks. We then take a step further to apply RankGAN in image captioning. We explore how to generate captions that are not only accurate in describing an image but also diverse across different images. By ranking human-written captions above image-mismatched captions within the image-caption joint space, the corresponding caption generator effectively exploits the inherent characteristics of human languages, and generates more diverse captions. In the second part, we focus on how to effectively edit inputs to generate new texts for specific natural language generation tasks, e.g., text style transfer and textual adversarial example generation. For the text style transfer task, we first examine the limitations and drawbacks of current generative models for text style transfer tasks with limited data. We then develop domain adaptive text style transfer models to leverage massively available data from other domains to solve the scarce data issue in the target domain. To generate textual adversarial examples, while previous rule-based editing methods are agnostic to the input context, we propose a contextualized perturbation approach to generate fluent and grammatical adversaries with better textual similarity. We further investigate three different perturbations to construct a richer range of generation strategies, resulting in a higher attack success rate of generated adversaries