Post-Desktop Interaction for Web Applications

Software applications are about to `leave' the desktop and to be used in a variety of different settings instead. When an application is developed, it cannot foreseen from which environments it will eventually be accessed. A good example for this are Web applications. They are accessed and used with many different Web browsers, depending on the user's choice. Many of the environments in which applications are accessed are becoming so-called post-desktop environments, i.e., environments that contain a variety of different interaction devices, beyond mouse, keyboard and screen, or even combinations of multiple interaction devices federated into a single UI. These environments are not necessarily equipped with a good network connection, so that the user may perceive disrupting latency due to the slow network connection, while interacting with applications The object of research of this thesis is the system infrastructure necessary to ensure that applications can be accessed in arbitrary post-desktop environments. We seek to augment and extend the Web system infrastructure to achieve this goal. This requires i) means to utilize novel interaction resources found in post-desktop environments, ii) coping with the dynamics of those environments, where resources are loosely coupled and iii) mitigating latency perceived by the user while interacting with Web applications in post-desktop environments. In pursuing this goal, the thesis makes the following contributions. In order to better understand the problem space, we developed a novel theoretical framework for describing the problem space of post-desktop access and use of applications. In order to handle the wealth and diversity of resources found in post-desktop environments one needs to be able to process input from arbitrary resources and to modify the output presented to the user at the same time. Existing approaches only support one of these two features at the same time. They are either optimized towards processing of input, in which case they rely on a pipeline architecture but then do not provide means for structured output modification. Or, they are optimized for modifying output, in which case their input-handling capabilities are limited to mouse and keyboard. The approach proposed in this thesis overcomes these limitations. Our concept of interaction strategies as realized in the MundoMonkey system can flexibly handle input from arbitrary sources, thanks to the connection to the MundoCore middleware, and at the same time can perform modifications of the UI using the rich DOM of Web applications. The approach has been evaluated by implementing several interaction strategies for interesting post-desktop environments, e.g., a strategy for voice-based interaction with Web applications proved superior to state-of-the-art solutions in a user study. In federated post-desktop environments the user has to deal not only with the diversity of resources, but also with the dynamics of the environment. Resources appear and disappear; the accessed applications must be easy to control for the user. We provides a single functionally complete meta UI to the user, allowing her to manage and control the post-desktop environment, which is not possible with existing approaches. The MineManager framework developed in this thesis is the first to enable the use of a digital meta UI in synchronization with physical changes to the access environment. An instance of this framework, called MineExplorer, has been implemented for the iPhone and the MundoCore middleware. It has been evaluated with user tests that showed the validity of the underlying concepts. The problem of user-perceived latency has been addressed for Web applications before. However, MundoProxy which is based on the concepts developed in this thesis is the only solution that can be used in mobile access environments, and which additionally does not require any changes to the implementation of the back-end services or the Web application. Application independence is achieved by using intelligent prefetching algorithms that predict future user requests based on observed previous requests. The peculiarities of mobile access environments are addressed by the piggybacking mechanisms that enable efficient use of the network bandwidth as well as of the energy of the mobile device battery. The evaluation shows that the user-perceived latency can be reduced up to $25\%$ with our approach. Finally, we propose MundoWeb, which integrates the three components explained above into a conservative extension to the Web access system addressing the requirements of post-desktop environments