Cognitive Validation of a Computer-Based Assessment of Problem Solving: Linking Cognitive and Online Processes

In this study we tested a novel cognitive validation strategy that yoked participants ’ verbal protocols with their clickstream data using a problem solving assessment (IMMEX—Interactive MultiMedia EXercises). Participants were presented with a scenario and provided with relevant and irrelevant information to solve the task. Participants could access the information in any order and attempt to solve the problem at any time. The most frequently occurring cognitive processes were paraphrasing text, making accurate cause-effect inferences, and monitoring of problem solving behavior. Productive processes were related to success and consistent with scientific reasoning behavior on the task and unproductive processes were related to unsuccessful performance and consistent with poor reasoning. We found strong evidence of the cognitive validity of the IMMEX task. Components of an online process-based assessment testbed are identified. Embedded Assessment of Complex Skills Engineering education has been a leader in the novel use of computer-based instruction. The intended use of many of the applications is to increase students ’ understanding of the content an