Understanding and Critiquing Multi-Modal Engineering Design Explanations

Designers often use a series of sketches to explain how their design goes through different states or modes to achieve its intended function. Instructors find that learning how to create such explanations is a difficult problem for engineering students, thus giving the impetus to create a design coach which allows students to practice explaining their designs and give feedback on said explanations. Given the complexity and wide scope of engineering design sketches, creating this design coach is a challenging AI problem. When communicating with sketches, humans act multi-modally, using language to clarify what would often be ambiguous or crude drawings. Because these drawings can be ambiguous even to human viewers, and because engineering design encompasses a very large space of possible drawings, traditional sketch recognition techniques, such as trained classifiers, will not work. This dissertation describes a coaching system for engineering design, CogSketch Design Coach. The claims are that (1) descriptions of mechanisms, specified multi-modally using sketches and restricted natural language must be understood via qualitative reasoning, and (2) the validity and clarity of an explanation of a mechanical design can be evaluated using a combination of design teleology and qualitative reasoning. These claims are supported by a set of evaluations of the system on sketched explanations. The work results in the following contributions: (1) new extensions to qualitative mechanics, a qualitative model of physics reasoning, for use with sketched input, (2) two new algorithms for identifying spatial mechanical relationships, (3) two new algorithms for generating items for critique of engineering design explanations, and (4) a teleology for describing and critiquing explanations of engineering designs. Through a classroom intervention in a first year engineering design course the work resulted in (5) a corpus of 240 multi-modal explanations and (6) data on a phenomenon we describe as sketching anxiety. These contributions form a set of resources for building new engineering design tools like CogSketch Design Coach and provide insight into future challenges for AI and intelligent coaching systems for classroom settings