Addressing Class Imbalance in Grammatical Error Detection with Evaluation Metric Optimization

We address the problem of class imbalance in supervised grammatical error detection (GED) for non-native speaker text, which is the result of the low proportion of erro-neous examples compared to a large num-ber of error-free examples. Most learn-ing algorithms maximize accuracy which is not a suitable objective for such imbal-anced data. For GED, most systems ad-dress this issue by tuning hyperparame-ters to maximize metrics like Fβ. Instead, we show that learning classifiers that di-rectly learn model parameters by optimiz-ing evaluation metrics like F1 and F2 score deliver better performance on these met-rics as compared to traditional sampling and cost-sensitive learning solutions for addressing class imbalance. Optimizing these metrics is useful in recall-oriented grammar error detection scenarios. We also show that there are inherent difficul-ties in optimizing precision-oriented eval-uation metrics like F0.5. We establish this through a systematic evaluation on multi-ple datasets and different GED tasks.