Extracting References Between Text and Charts via Crowdsourcing

News articles, reports, blog posts and academic papers of-ten include graphical charts that serve to visually reinforce arguments presented in the text. To help readers better under-stand the relation between the text and the chart, we present a crowdsourcing pipeline to extract the references between them. Specifically, we give crowd workers paragraph-chart pairs and ask them to select text phrases as well as the cor-responding visual marks in the chart. We then apply auto-mated clustering and merging techniques to unify the refer-ences generated by multiple workers into a single set. Com-paring the crowdsourced references to a set of gold standard references using a distance measure based on the F1 score, we find that the average distance between the raw set of ref-erences produced by a single worker and the gold standard is 0.54 (out of a max of 1.0). When we apply clustering and merging techniques the average distance between the unified set of references and the gold standard reduces to 0.39; an improvement of 27%. We conclude with an interactive doc-ument viewing application that uses the extracted references; readers can select phrases in the text and the system highlights the related marks in the chart