Developing characterizations of problem-solving processes, strategies, and challenges from process and product data in digitally delivered interactive assessments: case study.

Games and simulation-based assessments (GSBAs) are the focus of increased interest in educational assessment given their ability to operationalize assessment tasks that mimic real world scenarios. Combined with the capacity to unobstrusively collect data on task-solving behavior, sometimes referred to as \emph{process} or \emph{event} data, GSBAs have the potential to expand the scope and nature of inferences about students' skills, knowledge, and abilities. A case study and a simulation study explored the viability of using concepts and analytical tools from the field of Business Process Mining (BPM) to facilitate the generation of evidence identification rules from behavioral, event-based data generated in the context of a GSBA. The case study demonstrate the utility of a process guided by the principles of Evidence-Centered Design (ECD) in order to define and refine Student, Task and Evidence Models. The BPM conceptual and analytical tools allowed to economically investigate the feasibility of using aspects of task-solving behavior, such as differences in targeted event sequences, as evidentiary sources. Bayesian Networks were then use to aggregate traditional score data with behavioral data in order to predict student membership to latent classes. Given the novel nature of the analytical method used to identify evidence rules, known as the Fuzzy Miner, a simulation study investigated the impact of sample size, expert classification of a training sample, behavioral variability, and modeling parameters in the ability of the method to identify differences in process structure across groups. The simulation results show that the method's robustness to several sources of noise, suggesting its utility as an exploratory tool to be integrated with expert judgment when generating evidence identification rules