A Study of Automated Essay Scoring Frameworks on Evaluating Malaysian University English Test Essays Based on Syntactic and Semantic Features

An Automated Essay Scoring (AES) system can use a trained computational model to evaluate an essay as close to the grade that a human rater would assign. The purpose of this study is to examine the performance of different machine learning methods in predicting Malaysian University English Test (MUET) essay grade based on syntactic features and semantic features and generalize frameworks accordingly. Based on the results, we found that syntactic features of an essay have a higher effect than semantic features towards essay grades. Besides, we also found that the differences between machine learning and deep learning algorithms were not obvious, and neither algorithm's performance can be considered excellent because the quadratically weighted Kappa (QWK) scores were less than 0.75. Instead of using any available public essay datasets, five MUET essay datasets were collected locally for this study, and we found that all datasets suffer from imbalanced grade distribution. Therefore, QWK score is preferred over accuracy as the standard evaluation metric for AES because it provides more valuable information when dealing with imbalanced datasets. To overcome the problem of imbalanced grade distribution, a resampling method called Synthetic Minority Oversampling Technique (SMOTE) is applied to the dataset to study the impact of the resampling method on the performance of the AES framework. However, the SMOTE resampling method has been found to degrade predictive model accuracy and QWK scores. In addition, this study also developed an e-learning platform called UNIMAS DBRater, which is currently used by UNIMAS pre-university English classes, and more and more local educational institutions have expressed interest and willingness to join this e-learning platform