Combining distant and partial supervision for relation extraction

Broad-coverage relation extraction either requires expensive supervised training data, or suffers from drawbacks inherent to distant supervision. We present an ap-proach for providing partial supervision to a distantly supervised relation extrac-tor using a small number of carefully se-lected examples. We compare against es-tablished active learning criteria and pro-pose a novel criterion to sample examples which are both uncertain and representa-tive. In this way, we combine the ben-efits of fine-grained supervision for diffi-cult examples with the coverage of a large distantly supervised corpus. Our approach gives a substantial increase of 3.9 % end-to-end F1 on the 2013 KBP Slot Filling evaluation, yielding a net F1 of 37.7%.