Methods for ordinal peer grading

Massive Online Open Courses have become an accessible and affordable choice for education. This has led to new technical challenges for instructors such as student evaluation at scale. Recent work has found ordinal peer grading, where individual grader orderings are aggregated into an overall ordering of assignments, to be a viable alternate to traditional instructor/staff evaluation [21]. Existing tech-niques, which extend rank-aggregation methods, produce a single ordering as out-put. However, a single ordering alone may not be sufficient for instructors to con-fidently determine the final grades. For instance, instructors would like to have an estimate of the uncertainty of each assignment’s grade as well. In this work, we tackle this problem by applying Bayesian techniques to the ordinal peer grading problem; in particular we use MCMC-based sampling techniques in conjunction with the Mallows model. Experiments are performed on real-world peer grading datasets along with an analysis of the quality of the learned posterior distributions.