Leveraging Large Language Models for the Generation of Programming Exercises in Higher Education

Lecturers are increasingly attempting to use large language models (LLMs) to simplify and make the creation of exercises for students more efficient. This paper explores the potential of using LLMs such as GPT-3.5-turbo, GPT-4, and LaMDA to automatically generate programming exercises for use in higher education. Using applications such as ChatGPT, Bing AI Chat, and Google Bard, we generate exercises and assess the LLM’s suitability for generating a wide range of programming exercises. Results show that while LLMs can provide initial exercise designs and novel question structures, manual refinement is essential to ensure accuracy and coherence. We observe common error patterns across these LLMs that highlight certain challenges in understanding complicated programming concepts. Conversely, certain LLMs exhibit strengths in different exercise topics, highlighting the differences between these models. In summary, this research highlights the value of LLMs in generating exercises and emphasizes human supervision’s critical role in refining these exercises