Effects of Expertise and Strategies on Program Comprehension in Maintenance of Object-Oriented Systems: A Controlled Experiment with Professional Developers

The research presented in this thesis identifies the comprehension strategies used by professional programmers with different expertise during maintenance tasks on a previously unknown application in Java. Furthermore the effects of the strategies on performance and the effects of expertise on strategies and performance are investigated. The skill to comprehend a program is useful, especially in relation to software maintenance. In order to improve software maintenance, knowledge of how professional programmers comprehend a program is important. This knowledge can be used to develop techniques to support the process, or to improve education. We conducted a controlled experiment with 24 professional developers from five companies. The participants were introduced to a 3600 LOC Java application on which to perform three maintenance tasks. Their actions were logged and we have developed a data analysis tool to get a detailed picture of the participants’ actions. Our further development of GRUMPS made it possible to see the participants’ use of documentation, classes visited, when they compiled and how they executed the application. The data analysis tool can be reused by researchers in similar studies and contribute to improve the understanding of developers’ program comprehension strategies. In addition to the GRUMPS data, feedback was collected through the feedback-collection method. Two different approaches to the task solving were identified. In the first one, called systematic strategy, the participants tried to get an overview of the application before performing the maintenance tasks, by reading the system documentation or running the application. 75% of the participants applied this strategy. In the second one, called as-needed strategy, the participants went straight to the task solving, without trying to acquire any knowledge of the system on which to perform the tasks. This strategy was applied by 25% of the participants. We compared the performance of the participants applying the two strategies with respect to the correctness of solution, use of documentation, use of compilation and execution facilities and the solution time. The assessment of the solutions showed that there was a correlation between modifying Task 2 successfully and using the systematic strategy. However, in Task 3 and Task 4 those who used the as-needed strategy also performed well. The participants who used the systematic strategy spent more time solving Task 2, but for Task 3 and Task 4 the results were reversed. Three levels of programming expertise were identified and from the total number of 24 participants, eight participants were object-oriented novice, four participants were procedural expert and twelve participants were object-oriented expert. It was a clear difference in the comprehension approaches between the three groups of participants. The as-needed strategy was applied most by the object-oriented novices who also compiled more. These participants showed weaker results both in Task 2 and Task 4. Thus, novices should learn from the experts in order to maintain a system successfully