From Direct manipulation to Gestures: Moving the Expressive Power from the Displays to the Fingers

Optimizing the bandwidth of the communication channel between users and the system is fundamental for designing efficient interactive systems. Apart from the case of speech-based interfaces that rely on users’ natural language, this entails designing an efficient language that users can adopt and that the system can understand. My research has been focusing on studying and optimizing the two fol- lowing types of languages: interfaces that allow users to trigger actions through the direct manipulation of on-screen objects, and interactive systems that allow users to invoke commands by performing specific movements. Direct manipulation re- quires encoding most information in the graphical representation, mostly relying on users’ ability to recognize visual elements; whereas gesture-based interaction interprets the shape and dynamics of users’ movements, mostly relying on users’ ability to recall specific movements. This manuscript presents my main research projects about these two types of language, and discusses how we can increase the efficiency of interactive systems that make use of them. When using direct manip- ulation, achieving a high expressive power and a good level of usability depends on the interface’s ability to accommodate large graphical scenes while enabling the easy selection and manipulation of objects in the scene. When using gestures, it depends on the number of different gestures in the system’s vocabulary, as well as on the simplicity of those gestures, that should remain easy to learn and execute. I conclude with directions for future work around interaction with tangible objects