Targeted Syntactic Evaluation of Language Models

We present a dataset for evaluating the grammatical sophistication of language models (LMs). We construct a large number of minimal pairs illustrating constraints on subject-verb agreement, reflexive anaphora and negative polarity items, in several English constructions; we expect LMs to assign a higher probability to the grammatical member of each minimal pair. An LSTM LM performed poorly in many cases. Multi-task training with a syntactic objective improved the LSTM’s accuracy, which nevertheless remained far lower than the accuracy of human participants. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in an LM