Automated Essay Scoring by Maximizing Human-machine Agreement

Previous approaches for automated essay scoring (AES) learn a rating model by min-imizing either the classification, regression, or pairwise classification loss, depending on the learning algorithm used. In this paper, we argue that the current AES systems can be further improved by taking into account the agreement between human and machine raters. To this end, we propose a rank-based approach that utilizes listwise learn-ing to rank algorithms for learning a rating model, where the agreement between the hu-man and machine raters is directly incorpo-rated into the loss function. Various linguistic and statistical features are utilized to facilitate the learning algorithms. Experiments on the publicly available English essay dataset, Au-tomated Student Assessment Prize (ASAP), show that our proposed approach outperforms the state-of-the-art algorithms, and achieves performance comparable to professional hu-man raters, which suggests the effectiveness of our proposed method for automated essay scoring.