Breaking the Binary: Evaluating Gender Inclusivity in Neural Coreference Algorithms

The task of coreference resolution algorithms is to, given a text document, detect spans of word that refer to the same person, or entity. Recent development within the field of coreference resolution has focussed on methods employing neural networks and deep learning (Clark & Manning 2016, Lee et al. 2017, 2018, Barhom et al. 2019, Joshi et al. 2019, Kirstain et al. 2021). One key consideration in the development of these language processing models is how their output might be biased with regards to gender. Prior research has established that coreference models, like many other applications of natural language processing, exhibit bias across the binary masculine-feminine gender divide (Rudinger et al. 2018; Webster et al. 2018; Stanovsky et al. 2019; Levy et al. 2021; Savoldi et al. 2021). However, the bias of these systems in transgender, gender-nonbinary, and genderqueer contexts is not well-studied (Cao & Daumé III, 2020). In this study, I use the gender-inclusive GICoref dataset (Cao & Daumé III, 2020) to evaluate the performance of recent neural coreference models in a gender-inclusive setting. I demonstrate that the gender-inclusivity of coreference algorithms has indeed improved over the last seven years, although they still underperform in gender-inclusive settings compared to the standard test environments