Scoring Serious Educational Games Fairly

Abstract: Many people would like to see new skills, such as managing systems or complex problem solving, introduced into mainstream education. These skills are in high demand in the workplace. However, education is still an assessment-driven environment, and until these skills can be tested fairly, their impact will be minimal. Online games provide a way of evidencing this kind of ability, but games scores do not currently meet requirements of fairness. In high stakes assessment, 'fairness' is a complex concept, but it always has to have a mathematical argument behind it, for ethical and legal reasons. The analysis techniques that assessors use to locate problems in large data sets have been applied to a range of testing scenarios, from multiple choice to human-scored evaluation of complex tasks, but they will not work with game data at the moment. Bayesian analysis appears to be the best approach to deal with the more complex behaviour produced during game play, but few assessors have worked with Bayes probability. This talk will give you an insight into how assessors mathematically model very common human test behaviours, such as cheating, guessing, a rogue examiner or poor question design. It will also outline how key assumptions about testing need to be re-conceptualised for game data, and suggest how existing approaches to identifying bias and error might be incorporated in Bayesian probability-based estimations of ability. Background of Speaker : Clare Walsh is a teacher, and final year PhD student. Before joining the CDT, she authored over 20 course books that are used in secondary schools and further education worldwide, and has worked for over 15 years in international high stakes assessment design