The EmVoc framework for empirically evaluating modeling language vocabulary qualities: a pilot evaluation study

Conceptual modeling is central for a variety of activities surrounding the planning, design, development and maintenance of software-intensive systems. To allow development of conceptual models that are understood in the same way by different people, conceptual modeling languages are developed which contain concepts and rules that dictate how correct models can be built according to the language. A key component of a modeling language is a set of concepts that modelers must use to describe world phenomena. Once the concepts are chosen, a visual and/or linguistic vocabulary is adopted for representing the concepts. Both the choices of concepts and the vocabulary used to represent them, however, may affect the quality of the language under consideration. Some choices may match the intentions of the language or may allow for a shared understanding of the concepts better than other choices. We present EmVoc, a framework for empirically measuring the appropriateness of conceptual modeling language vocabularies based on observing how language user sample classify domain content under concepts of the language. The framework is based on a set of abstract empirical constructs that show how such empirical observations can be analyzed in order to detect vocabulary issues of various kinds. The constructs can be operationalized into concrete measures based on the specific data collection instrument or interests of the study. As a first evaluation of our framework we apply it to compare an existing language with an artificial one that is manufactured to exhibit specific issues. We then test if the metrics indeed detect these issues and only. In this paper, we present the complete experimental design and report on the results of a pilot administration. The complete study will be reported in a future version of this paper