Understanding Illustrations of Physical Laws by Integrating Differences in Visual and Textual Representations

An important problem in the integration of vision and language is comprehending explanatory diagrams, such as those found in science and engineering textbooks. One class of diagrams, which we call juxtaposition diagrams, illustrate a physical principle by comparing two similar situations that vary in a carefully chosen way. This paper describes research in progress on a computational model, JUXTA, which analyzes juxtaposition diagrams. JUXTA performs its analysis by finding the interesting differences in a figure, and then relating those differences to differences stated in the diagram caption. By using the visible differences in the figure as reference points for the qualitative relationship given in the caption, JUXTA is able to intelligently label the relevant parts of the figure. JUXTA also critiques the figure for understandability, warning of differences in the figure which may confuse the reader, and noting visible differences in the figure which are irrelevant and may be removed. 1