Neural NLP models under low-supervision scenarios

Neural models have been shown to work well for natural language processing tasks when one has large amounts of labeled data, but problems arise when this is not the case. In this thesis we investigate several ‘low-supervision’ scenarios in which we do not have sufficient training data, and we propose methods to improve performance in these scenarios. First, we consider the scenario where we can use other types of resources in addition to the limited training labels. For instance, we can ask human annotators to provide rationales supporting their labels (annotations) for training examples. To capitalize on such supervision, we develop a neural model that can train on both instance labels and associated rationales. We also investigate how to incorporate existing ontologies into neural models. Specifically, we develop a novel training algorithm that enforces weight sharing among similar words in the ontologies, thus inductively biasing the neural model training. In addition incorporating other types of resources beyond instance labels, we also use transfer learning techniques which are general means of learning in low-supervision settings. We study how to use multiple sets of pre-trained word embeddings as inputs to neural models, and fine-tune them to the task at hand in a more intelligent way than simply concatenating them. We also develop a novel model for text generation, in which the model is able to generate text from a new domain (unseen in training data). Rather than simply fine-tuning the model on the target domain, the model fully uses the domain information in the training set, allowing it to generate domain specific text. Lastly, we consider how to collect data under a limited budget more efficiently than simply random selection of unlabeled data for annotation. We develop new active learning (AL) methods to collect more informative examples to be annotated specifically for neural models, so that better models and more discriminative text representation can be learned with fewer labels. Following this, we further develop new AL approaches when we have richly annotated data from a relevant domain, that is, we combine AL and transfer learning and leverage the advantages of both methods. We also investigate how to use the pre-trained deep bidirectional transformer (BERT) to actively select labels.Computer Science