Perceptual-cognitive Properties of Pictures, Diagrams, and Sentences: Toward a Science of Visual Information Design

Right now you are reading a sentence. Earlier, you might have been looking at a realistic picture, such as a photograph, or an outline drawing in a set of instructions. If you are a programmer, you work with sentence-like structures, such as code, or a system diagram. These are all graphic representations. To varying degrees, the effectiveness of every graphic representation relies on its ability to convey the designer’s intended meaning and elicit the intended reaction from its audience. However, the design of graphic representations, even in technical domains such as visual programming language design or interactive information visualization, currently relies heavily on general principles based solely on practice, intuition, and informal measures of effectiveness from the applied art and craft of design (as opposed to scientific analysis or theory). There is an increasing demand for a scientific understanding of design and its evaluation from stakeholders (who seek evidence for effectiveness) and designers (who seek to advance their field). Because both the creation of graphic displays and their perception are literally embodied experiences, a model was developed with an embodiment orientation, specifically based on how graphics are perceptually and cognitively processed. In my research, I found that graphic representations are constituted of two properties, pictorial and symbolic information, that emerge through two interrelated aspects of perception. In sighted individuals, for example, every graphic representation makes use of biological capabilities to process visual sensation (i.e., light hitting the retina), which are processed in relation to culturally-learned capabilities (i.e., writing). I observed how graphic representations – such as pictures, diagrams, and sentences – are “naturally selected” (i.e., during different phases of design or problem solving). From these observations, I developed a model that distinguishes and predicts the effectiveness of pictures, diagrams, and sentences, in terms of how object relations and attributes are pictorially or symbolically represented, relative to the functional roles of those representations, contexts, and in some cases, individual perceptual-cognitive differences among perceivers. This model is a step toward a science of graphics that could lead to evaluation techniques for information systems, theories for inclusive design, and ergonomically designed software programming tools.Ph