Robust Data-to-text Generation with Pretrained Language Models

The task of data-to-text generation amounts to describing structured data, such as RDF triples, in fluent natural language sentences. The state-of-the-art approach in research systems today is finetuning pretrained language models (PLMs). This often leads to overfitting the data and may produce hallucinations, i.e. situations where the PLM generates outputs that are not grounded in the input, typically replicating (or amplifying) training data noise. Rather than applying a PLM as black box for the whole data-to-text task, we aim at using PLMs for simple individual subtasks, aiming to achieve broad generalization and minimize hallucination. First, we use a pipeline approach where the PLMs only work as text "editors", rather than generators, taking advantage of their high output fluency. The data is converted into text in an initial preprocessing step, where we use simple handcrafted templates (one per RDF relation). This results in very short sentences recounting the individual input facts, corresponding to RDF triples. The PLMs then order the individual facts and fuse them into fluent sentences. This helps us generate without in-domain training data (except the simple templates) and achieve good fluency and accuracy. We further examine the capability of PLMs to produce accurate descriptions of individual RDF triples, in order to remove the last handcrafted step. Using a specially collected dataset of varied RDF triple descriptions, we show that PLMs finetuned with a variety of relations are very robust in verbalizing novel, unseen relations. The key to PLMs' usability here is providing clear and meaningful relation labels