Reliability and Validity Evidence for an Object Assembly Test of Engineering Sketching.pdf

Sketching is a valuable skill in engineering for representing information, developing design ideas, and communicating technical and abstract information. Design thinking is supported through sketching as a means of translating between internal and external representations, and creating shared representations of collaborative thinking. Sketching is also an important means of developing spatial abilities which are predictive of success in STEM fields. Computer-based design visualization tools have largely replaced freehand sketching in undergraduate engineering classrooms, but the shift has negatively impacted students’ design thinking and spatial reasoning skills. While many published classroom assessments of engineering and engineering design sketching skill exist, few are linked to theory of mental rotation and mental imagery, and the validity evidence for these instruments is scarce. This dissertation reports the development of a new instrument to assess sketching skills in engineering education based on spatial reasoning skills. The first study is a systematic literature review of engineering education literature on sketching assessment, sketching constructs and metrics were identified across existing tests, as well as cognitive theory which informed their use and wider learning contexts and purposes for sketching assessment. From content analysis after abstract and full paper sorting and review, metrics on accuracy, perspective, line quality, annotations, and aesthetics were found to be most prevalent. Cognitive skills included perceiving the sketch subject, creatively sketching ideas, using metacognition to monitor the sketching process, and communication. Sketching assessment varies by discipline and relies on feedback and scores or grades as well as expert review. From these findings, a new Object Assembly Test of Sketching was developed to evaluate sketching skills on 3-dimensional mental imagery and mental rotation tasks in 1- and 2-point perspective. The second study describes two rounds of pilot testing and revisions to the Object Assembly Test over Fall 2021 and Spring 2022 in two sections of an undergraduate mechanical engineering course. Initial inter-rater reliability values were low and rubric categories had inconsistent performance, and mean scores suggested line quality metrics were more difficult for students than shape quality metrics. Revisions to rubric categories were made after consulting with subject matter experts in engineering design, industrial design, and assessment, and a second round of pilot testing showed improvement in reliability between raters with similar patterns of mean scores. The third study presents reliability and preliminary validity evidence for the Object Assembly Test’s use in undergraduate mechanical engineering design graphics courses. Correlation and repeated-measures ANOVA were used to investigate pre-post score differences before and after sketching classroom learning intervention. A linear relationship between Object Assembly scores and intelligent tutoring software sketching metrics was also modeled for pre-post scores. Inter-rater reliability was improved through two rounds of grading and discussion. Correlations were moderately positive between scores and metrics, with more complex exercises negatively related to Speed. Post-test scores were significantly higher than pre-test scores but did not interact significantly with skill. Linear regression results showed some significant prediction of Precision and Smoothness by shape quality metrics, and a clear speed-accuracy tradeoff with negative prediction of Speed by nearly all sketching skills. We anticipate future use for this instrument where instructors and researchers may implement the sketching exercises and rubric in engineering classrooms alongside 3-dimensional drawing software. The Object Assembly Test can provide students with opportunities to practice perspective sketching before using computer design tools, as well as apply mental imagery and mental rotation cognition when manipulating complex solid shapes for sketching. Ongoing validation studies with this instrument will expand to a more diverse test-taking population and develop a theoretical model of mental rotation and mental imagery skills to inform object assembly sketching.