On multi-modal interfaces and the construction of conceptual models

In recent years, systems have begun to emerge that use speech in the user interface. However, compared with graphical user inlerfaces, speech-only interfaces have several problems: speech technology is not yet sufficiently robust. and it is more difficull to make clear to the user which funct ionality is available and how it can be accessed. We explore potential solutions to these problems by means of presenting a visual representation of the domain of discourse and the state of the dialogue. We describe an experiment in which a uni-modal (speech-only) and a multi-modal interface are compared in terms of effectiveness, efficiency, and satisfaction. The results show a strong learning effect. Subjects who first work with the multi-modal system and then switch to the uni-modal system have a strong advantage over subjects who start with the uni-modal system, switching to the multi-modal system later on. The results are discussed in terms of the need to establish an appropriate conceptual model of the system as early as possible