USING ANALOGY TO MODEL INTEGRATION DURING MULTIMODAL KNOWLEDGE CAPTURE

Learning from textbooks requires inte-grating knowledge from text and diagrams. Studies from cognitive and learning sciences have shown that these multimodal sources of information often lead to better performance on recall and transfer tests than single modality sources. This paper describes a cognitive model of multimodal knowledge capture that uses analogy to model the cognitive process of integrating information from text and diagram representations. We demonstrate our system using a chapter of a physics textbook, evaluat-ing its performance using the publisher-provided homework problems. We show how the Structure Mapping Engine can be used for both multimodal integration and problem solv-ing tasks