Annotation via LLM (Large Language Models) in Digital Literary Studies: beating (about) the bots?

As part of our experiments with doing Aspect-Category-Opinion-Sentiment Quadruple Extraction, in this talk, we aim to weigh some of the tried and trusted methods of doing NLP (BERT Transformers) against the new kid on the block, namely LLM (Large Language Models) trained for various purposes and languages. Few-shot and zero-shot approaches may seem adventurous, and they indeed may not live up to the state-of-the-art achieved with regard to our specific use-case, i.e. aspect-based sentiment analyses (ABSA) in a very specific domain (literary criticism). Nevertheless, we will compare the results of our BERT-based setup fine-tuned on annotations against a naive GPT-3-based few-shot approach. Preliminary results show that even the zero-shot approach yield surprisingly good results, especially for combining ABSA with NER (based on the real-world knowledge stored somewhere in the LLM). The accompanying GitHub repository is a PoC that shows how one may "escape" the dummy chatbox interface in order to feed data in bulk to the model. Whether this is a viable solution (in terms of cost and ethics) remains open for discussion. But especially for the aspiring DH researcher with limited programming skills, limited GPU access, and/or without persistent support from teams dedicated to humanities-style qualitative research questions, this can be an quick & dirty solution. The setup allows to overcome quite some barriers (multilingual sentiment in a specific multimodal setting, working with oral data via automatic transcription of oral sources, alignment of oral and written data, speaker diarization, automatic generation of training data with annotations, etc). We will evaluate pros and cons of the API-based solution for classification, such as the small amount of training data needed to fine-tune the model, the ease of access, and the cost. All in all, this type of modular pipeline is a less prohibitive pathway for aspiring researchers, compared to training and setting up a demanding and complicated environment that is typically required in NLP methods. In addition, allowing people to "chat with your data" will inevitably become a new way of disseminating research results in a non-directive way, also in domains where access to the actual artefacts restricted by copyright.As part of our experiments with doing Aspect-Category-Opinion-Sentiment Quadruple Extraction, in this talk, we aim to weigh some of the tried and trusted methods of doing NLP (BERT Transformers) against the new kid on the block, namely LLM (Large Language Models) trained for various purposes and languages. Few-shot and zero-shot approaches may seem adventurous, and they indeed may not live up to the state-of-the-art achieved with regard to our specific use-case, i.e. aspect-based sentiment analyses (ABSA) in a very specific domain (literary criticism). Nevertheless, we will compare the results of our BERT-based setup fine-tuned on annotations against a naive GPT-3-based few-shot approach. Preliminary results show that even the zero-shot approach yield surprisingly good results, especially for combining ABSA with NER (based on the real-world knowledge stored somewhere in the LLM). The accompanying GitHub repository is a PoC that shows how one may "escape" the dummy chatbox interface in order to feed data in bulk to the model. Whether this is a viable solution (in terms of cost and ethics) remains open for discussion. But especially for the aspiring DH researcher with limited programming skills, limited GPU access, and/or without persistent support from teams dedicated to humanities-style qualitative research questions, this can be an quick & dirty solution. The setup allows to overcome quite some barriers (multilingual sentiment in a specific multimodal setting, working with oral data via automatic transcription of oral sources, alignment of oral and written data, speaker diarization, automatic generation of training data with annotations, etc). We will evaluate pros and cons of the API-based solution for classification, such as the small amount of training data needed to fine-tune the model, the ease of access, and the cost. All in all, this type of modular pipeline is a less prohibitive pathway for aspiring researchers, compared to training and setting up a demanding and complicated environment that is typically required in NLP methods. In addition, allowing people to "chat with your data" will inevitably become a new way of disseminating research results in a non-directive way, also in domains where access to the actual artefacts restricted by copyright.