Measuring the Understandability of Business Process Models - Are We Asking the Right Questions?

In this paper, we show how experiments on the understandability of business process models can depend on the exact wording used in the experiments’ questionnaires. For this purpose, we partially replicated a published experiment. We asked a group of students a number of questions on relations between tasks in a business process model. Alternatively, we used a set of modified questions which were aimed to ask for exactly the same relations. The result was that there was a significant difference in the number of correct answers between the two systems to construct a question. We argue that a non-negligible part of the wrong answers given in the experiment did not result from problems to understand the model, but rather from problems to understand the question. It follows that it is dangerous to draw conclusions from such an experiment until enough effort has been taken to select appropriate questions