How plans occur in novices' programs: A method to evaluate program-writing skills

Program-writing skills are often assessed with computer-based practical exams where students are required to design, write and test small programs according to specifications written in natural language. The evaluation of such tasks is usually based on the correctness of outputs (possibly with the support of automatic tools) and the quality of code. In this paper, we propose to evaluate program-writing assignments by using Soloway's notion of plans [34]. We illustrate this idea by defining a marking scheme for some typical program-writing tasks and apply this approach to evaluate the programs written during the final examination of an introductory programming course. We then compare the resulting marks with the marks originally assigned by the instructors of the course. Findings show that the use of plans in the marking process is feasible and requires an effort similar to the one required by a traditional evaluation process, and the focus on plans when examining the source code of students gives useful insights about their difficulties