CoMoVA - A comprehension measurement framework for visualization systems

Despite the burgeoning interest shown in visualizations by many disciplines, there yet remains the unresolved question concerning comprehension. Is the concept that is being communicated through the visual easily grasped and clearly interpreted? Visual comprehension is that characteristic of any visualization system, which deals with how efficiently and effectively users are able to grasp the underlying concepts through suitable interactions provided for exploring the visually represented information. Comprehension has been considered a very complex subject, which is intangible and subjective in nature. Assessment of comprehension can help to determine the true usefulness of visualization systems to the intended users. A principal contribution of this research is the formulation of an empirical evaluation framework for systematically assessing comprehension support provided by a visualization system to its intended users. To assess comprehension i.e. to measure this seemingly immeasurable factor of visualization systems, we propose a set of criteria based on a detailed analysis of information flow from the raw data to the cognition of information in human mind. Our comprehension criteria are adapted from the pioneering work of two eminent researchers - Donald A. Norman and Aaron Marcus, who have investigated the issues of human perception and cognition, and visual effectiveness respectively. The proposed criteria have been refined with the help of opinions from experts. To gauge and verify the efficacy of these criteria in a practical sense, they were then applied to a bioinformatics visualization study tool and an immersive art visualization environment. Given the vast variety of users and their visualization goals, it may be noted that it is difficult for one to decide on the effectiveness of different visualization tools/techniques in a context independent fashion. We therefore propose an innovative way of evaluating a visualization technique by encapsulating it in a visualization pattern where it is seen as a solution to the visualization problem in a specific context. These visualization patterns guide the tool users/evaluators to compare, understand and select appropriate visualization tools/techniques. Lastly, we propose a novel framework named as CoMoVA (Comprehension Model for Visualization Assessment) that incorporates 'context of use', visualization patterns, visual design principles and important cognitive principles into a coherent whole that can be used to effectively tell us in a more quantifiable manner the benefits of visual representations and interactions provided by a system to the intended audience. Our approach of evaluation of visualization systems is similar to other questionnaire-based approaches such as SUMI (Software Usability Measurement Inventory), where all the questions deal with the measurement of a common trait. We apply this framework to two static software visualization tools in the software visualization domain to demonstrate the practical benefits of using such a framework