Experiments towards digital exam with auto-grading in C++ programming courses

Climbing Mont Blanc (CMB) is an online judge system especially suited for evaluating energy efficient programming solutions currently in development by a team of professors and master students at the Norwegian University of Science and Technology (NTNU). This project seeks to further gain knowledge and experience in the field of automatic assessment of programming problems, by identifying various question types suitable for automatic and reliable assessment. We have found that most of the question types commonly found on learning management systems, such as itslearning and Blackboard, are suitable in an exam situation. As for CMB, we have found that \textit{complete the code} and \textit{fix the code} problem types can be considered suitable in most cases because the problem creator has much more control over the program flow. Problems that require complete implementations and topic specific programming problems are also suitable but in many cases measures needs to be done to maintain validity of the evaluation. To reach a conclusion on the feasibility of digital examination at NTNU, we have conducted a midterm experiment with the students in the TDT4102 course, which has been described in detail. Elements such as the execution, result extraction and anonymization of the respondents are covered. The latter was done using a representative from the CMB team who had no formal connection to the experiment in itself. All results from the midterm test were statistically analyzed by testing a set of null hypotheses that were defined early in the project. This analysis confirmed what we originally believed; the use of familiar digital tools such as the CMB system is beneficial when solving programming problems, compared to writing code by hand. Finally, we conclude that the midterm experiment was successful, due to the results from both the midterm and questionnaire indicated that the system has potential. We also suggest a plan for the project's continuation in the years ahead