The Role of Compilation Mechanisms in Novice Programming Behaviour

Compilation mechanisms, like many features in programming environments, are understudied in terms of their effects on users, especially in the case of programming students. In this thesis we explore how these mechanisms affect the programming behaviour of novices towards better understanding their roles in the programming process. We conduct our study through different compilation mechanism setups that are present in two major version releases of the BlueJ introductory pedagogical environment for the Java programming language. BlueJ 3 features ``manual compilation with enforced first error message presentation'', while BlueJ 4 supports ``automatic and manual compilation with on-demand any/all error message presentation''. We examine the programming behaviour of thousands of novices by utilizing programming process data mined from Blackbox, a large repository that stores programming-related information organically generated by BlueJ users. We begin our investigation by exploring how the differentiation in compilation mechanisms affects the interaction between the user and the programming environment in terms of quantity of displayed compiler error messages, manual compilations, and rates of manual compilation success. We found that novices tend to see more error messages, compile manually less frequently, while their manual compilations have higher success rates in BlueJ 4 over BlueJ 3. We also observed that novices' quantified interaction with BlueJ 3 follows a more deterministic distribution compared to the interaction of users with BlueJ 4, due to the environment restrictions. In addition, we found more intensified differences in the interaction with the two BlueJ versions when users are exposed to a single compilation mechanism than when exposed to multiple mechanisms, with users in the latter group showing a moderated interaction, possibly due to adapting their habits accordingly. Further, we explored the compilation behaviour of novices in terms of the syntax state of their source code and the trigger state of their compilations, the error resolution time, as well as their interaction with compiler error messages during the programming process. We identified multiple similarities and differences in compilation behaviour when using the different compilation mechanisms, and also observed lower error resolution times in BlueJ 4. As the interaction of users with error messages is inherently more complicated in BlueJ 4 than in BlueJ 3, we provide details on the manner that displayed error messages emerge while programming. As part of our investigation, we also conducted surveys with programmers with different experience levels that targeted the perceived usefulness of compilation mechanisms, and asked them to provide suggestions for improvement. We found that programmers perceive the error message presentation present in BlueJ 4 as more useful and their overall suggestions on improvement aligned more towards the functionality of the mechanisms present in BlueJ 4 than those in BlueJ 3. Finally, we conducted an observational study with postgraduate students in order to draw contextualized inferences and evaluate prior assumptions that derived from non-contextual data. Substantial differences in the manner that users interact with compilation mechanisms and approach the resolution of errors were observed, depending on how these mechanisms are facilitated within the environment. Based on our findings, we discuss the implications of the studied compilation mechanisms in the programming process, since substantial changes emerged in the behaviour of novices, and provide a set of design guidelines for programming environments. This thesis aims to inform the research community with an inquiry on mechanisms that influence the programming process of students, and provide information to pedagogical tool designers about the suitability of their design decisions for the first steps of learning how to program.2022-12-07 JG: Author's signature removed from PD