Methods for ordinal peer grading

Massive Online Open Courses have become an accessible and affordable choice for education. This has led to new technical challenges for instructors such as student evaluation at scale. Recent work has found ordinal peer grading, where individ-ual grader orderings are aggregated into an overall ordering of assignments, to be a viable alternate to traditional instruc-tor/staff evaluation [23]. Existing techniques, which extend rank-aggregation methods, produce a single ordering as out-put. While these rankings have been found to be an accurate reflection of assignment quality on average, they do not com-municate any of the uncertainty inherent in the assessment process. In particular, they do not to provide instructors with an estimate of the uncertainty of each assignment’s position in the ranking. In this work, we tackle this problem by ap-plying Bayesian techniques to the ordinal peer grading prob-lem, using MCMC-based sampling techniques in conjunction with the Mallows model. Experiments are performed on real-world peer grading datasets, which demonstrate that the pro-posed method provides accurate uncertainty information via the estimated posterior distributions