Criteria for evaluating usability evaluation methods

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard crite-ria for comparison. In this article, we present a practical discussion of factors, compari-son criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We high-light specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the princi-ples and techniques used to approach UEM evaluation and comparison. 1