Human-Machine Collaborative Annotation: A Case Study with GPT-3

Holter OM, Ell B. Human-Machine Collaborative Annotation: A Case Study with GPT-3. Presented at the LDK, Wien.Within industry, it is vital to adequately communicate the qualities and features of what is to be built, and requirements are important artefacts for this purpose. Having machine-readable requirements can enhance the level of control over the requirements, allowing more efficient requirement management and communication. Training a semantic parser typically requires a dataset with thousands of examples. However, creating such a dataset for textual requirements poses significant challenges. In this study, we investigate to what extent a large language model can assist a human annotator in creating a gold corpus for semantic parsing of textual requirements. The language model generates a semantic parse of a textual requirement that is then corrected by a human and then added to the gold standard. Instead of incrementally fine-tuning the language model on the growing gold standard, we investigate different strategies of including examples from the growing gold standard in the prompt for the language model. We found that selecting the requirements most semantically similar to the target sentence and ordering them with the most similar requirement first yielded the best performance on all the metrics we used. The approach resulted in 41 \% fewer edits compared to creating the parses from scratch, -- thus, significantly less human effort is involved in the creation of the gold standard in collaborative annotation. Our findings indicate that having more requirements in the gold standard improves the accuracy of the initial parses