Scaling Short-answer Grading by Combining Peer Assessment with Algorithmic Scoring

Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper integrates peer and machine grading to preserve the robust-ness of peer assessment and lower grading burden. In the identify-verify pattern, a grading algorithm first predicts a student grade and estimates confidence, which is used to es-timate the number of peer raters required. Peers then iden-tify key features of the answer using a rubric. Finally, other peers verify whether these feature labels were accurately ap-plied. This pattern adjusts the number of peers that evaluate an answer based on algorithmic confidence and peer agree-ment. We evaluated this pattern with 1370 students in a large, online design class. With only 54 % of the student grading time, the identify-verify pattern yields 80-90 % of the accu-racy obtained by taking the median of three peer scores, and provides more detailed feedback. A second experiment found that verification dramatically improves accuracy with more raters, with a 20 % gain over the peer-median with four raters. However, verification also leads to lower initial trust in the grading system. The identify-verify pattern provides an ex-ample of how peer work and machine learning can combine to improve the learning experience. Author Keywords assessment; online learning; automated assessment; peer learnin