Assessment of student problem-solving processes with interactive computer-based tasks

© 2009 Dr. Nathan Paul ZoanettiProblem solving is recognised as an important intellectual activity in schooling and beyond. In particular, generic problem-solving skills which transfer across learning areas are valued educational outcomes. The objective of this study was the design and evaluation of an online assessment system that provided diagnostic information on students' development of problem-solving competencies at upper primary and lower secondary school level. This resulted in the development of a methodology for collecting and interpreting problem-solving process data to assess important procedural aspects of problem solving. In this research study, existing assessment design and analysis methodologies were extended and applied to produce descriptions of problem-solving behaviour useful for both students and educators. The assessment system utilised recent advances in technology, assessment design and analysis, and problem-solving theory to guide the development of interactive computer-based tasks and to facilitate the interpretation of complex process data from student solution processes. Rules for interpreting computer-captured process data were empirically validated using qualitative verbal protocol analysis techniques. This study introduced a novel contribution to assessment design methodology called a temporal evidence map. This data transcription tool was designed for displaying and analysing concurrent sources of process data collected throughout task piloting exercises. Use of this tool culminated in the refinement of tasks and scoring rules, and informed development of additional tasks for the main data collection phase of the study. Following large-scale online data collection, the data were probabilistically modelled using Bayesian Inference Networks. A range of model evaluations were carried out to gauge aspects of assessment validity and reliability. Finally, the inferences generated via Bayesian modelling were used to produce diagnostic student profile reports suitable for informing instruction. Educators have much to gain from technology-based assessment systems underpinned by cognitively diagnostic models of cognition. In particular, supporting assessment inferences about procedural quality is well-aligned with 21st century skills in information-rich educational and vocational settings. This study provides diagnostic information to educators about how, and not just if, students solve problems