Developing domain-specific measure of performance for complex sociotechnical military systems

The selection of performance measures for the evaluation of complex systems is extremely difficult, frequently, there are multiple influencing factors, not always sensitive to system change. This paper proposes a new approach, using the output of Work Domain Analysis (the first phase of Cognitive Work Analysis), for selecting domain-specific interrelated measures of performance. This paper discusses the application of the approach to evaluate the effects of computerisation in a military land headquarters domain. Beliefs of subject matter experts were used to evaluate the impact of each of the measures of performance identified in the abstraction hierarchy. By evaluating the system at a number of levels of abstraction, and modelling the relationship between these levels; changes in high-level descriptions of system performance can be related to changes brought about by the introduction of new technology