The Evaluator Effect in Heuristic Evaluation: A Preliminary Study of End-users as Evaluators

Heuristic Evaluation (HE) is a popular usability inspection method. Yet little is known about the effect the evaluators have on the outcome of HE. One potentially important feature of evaluators is their end-user status, that is, whether or not they are end-users for whom the interface is designed. I completed a detailed review of the HE literature, combined sources, developed an explicit method for conducting an HE and trained HE novices from different work domains using it. Using these methods I conducted a preliminary randomized crossover study (n=6) of the effect of end-user status during the inspection and merging stages of HE. I estimate a larger study of approximately 148 end-users would be needed to test hypotheses regarding end-user status. I demonstrated a novel measure of the effect of end-user status for the merging stage of HE, which I called the measure of matching similarity (MMS).MAS