Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations

We study the problem of classifying the temporal relationship between events and time expressions in text. In con-trast to previous methods that require extensive feature engineering, our ap-proach is simple, relying only on a measure of parse tree similarity. Our method generates such tree similarity values using dependency parses as in-put to a convolution kernel. The re-sulting system outperforms the current state-of-the-art. To further improve classifier performance, we can obtain more annotated data. Rather than rely on expert annotation, we assess the feasibility of acquiring annotations through crowdsourcing. We show that quality temporal relationship annota-tion can be crowdsourced from novices. By leveraging the problem structure of temporal relation classification, we can selectively acquire annotations on problem instances that we assess as more difficult. Employing this anno-tation strategy allows us to achieve a classification accuracy of 73.2%, a statistically significant improvement of 8.6 % over the previous state-of-the-art, while trimming annotation efforts by up to 37%. Finally, as we believe that access to sufficient training data is a significant barrier to current temporal relationship classification, we plan to share our collected data with the re-search community to promote bench-marking and comparative studies.