Multimodal Design: An Overview

Design generally entails multiple kinds, or modalities, of representation and reasoning. For example, designers reason with different kinds of representations, including both imagistic (e.g., drawings, sketches and diagrams) and propositional (e.g., function, behavior, causality, and structure). This multimodal nature of design representation and reasoning raises several issues for AI research on design. For example, what types of knowledge are captured by various modalities of representation? What kinds of inferences are enabled and constrained by different representation modalities? How might we couple a representation in one modality with a representation in another, or transform a representation in one modality to another? AI has long been interested in these issues, though not necessarily in the context of design. AI research on multimodal representations and reasoning relevant to design has generally followed several important threads. In one thread, AI has sought to understand the various modalities in terms of the types of knowledge they capture and the inferences they enable. For example, Davis (1984) describes an early effort to declaratively represent and then reason about the structure and behavior of physical systems, an