A Framework for Constructing Visualization, Animation, and Direct Manipulation Interfaces

This thesis describes a framework for developing kinds of non-WIMP-based user interface software, designated as a bi-directional translation model. Here, 'non-WIMP-based'interface means that it cannot built only by combining WIMP widgets, such as buttons and menus. In particular, the targets of this thesis are two types of GUI software. One is direct manipulation interfaces for figures and diagrams that visually represent various abstract objects and relations in an application data, such as direct manipulation interfaces for network and hierarchical data structures. They enable the user to draw and modify diagrams to input abstract data into an application. Another is animations of changing abstract objects and relations in an application data, such as animations of sorting algorithms. They show their transitions visually, and are useful for understanding the behavior of an application or an algorithm. The bi-directional translation model models the general process of visualization, picture recognition, and animation generation. It is originated in Kamada's general framework for visualizing abstract objects and relations. We have extended it by integrating recognition of figures/diagrams and animation of changing abstract objects and relations, which is one of the contributions in this thesis. The key idea of the framework is that these functions are translations between different data representations. The model introduced four data representations: AR (Application Data Representation), ASR (Abstract Structure Representation), VSR (Visual Structure Representation), and PR (Pictorial Representation). In the model, visualization is a translation from AR to PR via ASR and VSR. Recognition of figures is an inverse translation from PR to AR via VSR and ASR. The essential translation is that between ASR and VSR, because there is no a priori mapping between these representations. ASR is the representation in our model that represents the structure of application data, and it does not have explicit visual appearance information. On the other hand, the purpose of VSR is to represent the high-level structures of pictures. The mapping between ASR and VSR is the essential conversion between abstract data for an application and pictorial data for presentation. Nevertheless, the structure of ASR and that of VSR are usually similar. As users understand the meaning of abstract application data via pictures, visualized pictures should represent abstract application data, and thus they should have a similar structure. The programmer specifies these translations by visual mapping rule sets, which are declarative rules that define mappings between ASR data and VSR data. As both structures are similar, mapping rules are usually simple. Animations are also handled in the framework extended naturally to the time dimension. In the framework, animations are regarded as operations on PR that are translated from operations on AR via operations on ASR and VSR. The translation among operations is executed maintaining consistency with the mapping relations among the AR, ASR, VSR, and PR data. We have chosen an interpolation-based method for implementation of the extended framework for animations. That is, animations are generated by interpolating a sequence of pictures translated from the running application's internal data. Rather than specifying animations (motions or transformations) directly, the programmer specifies transitional operations, i. e., the methods used for interpolating two pictures before and after the invocation of an operation, with transition mapping rules. Therefore, animations are determined by two types of mapping rules: visual mapping rules and transition mapping rules. In addition, we describe the systems implemented based on the framework. TRIP2 is a system that achieved the bi-directional translation between AR and PR. We applied TRIP2 to produce several examples such as the Othello application. It is implemented on NextStep using Objective-C and Prolog. To build direct manipulation interfaces using TRIP2, the programmer writes visual and inverse visual mapping rules in Prolog. TRIP2a is a tool for constructing abstract animations that depict the behavior of program executions. It is implemented by extending TRIP2 so that we can use two functions together: bi-directional translation between abstract data and pictures, and translations from abstract data into animations. The programmer can use the same visual mapping rules of TRIP2 for TRIP2a. TRIP2a/3D is the successor of the TRIP2a system that specializes in generating animations and can also handle three-dimensional representations and event-driven animations, which is useful for animating parallel program executions. In order to be able to view animations on various platforms, the animation viewers are separated from the translation module in TRIP2a/3D. The programmer writes visual mapping rules for each animation in KLIC which are compiled with translation modules to generate translators that output animation data from the log data of the application's execution. Generated animation data can be viewed with viewers on various platforms such as X-Window, MS-Windows, and Java. We describe various algorithm animations and visualizations of program executions generated with these systems. Supporting the debugging of visual mapping rules with our systems is difficult. As a step toward solving this problem, we describe techniques for visualizing constraint systems in visual mapping rule sets. One way is to show constraint systems as three-dimensional graphs. By looking at constraint graphs, the user can see the structure of constraint systems more directly than in their textual form. Another way is to animate constrained graphical objects to show their degrees of freedom in constraint systems by using cartoon techniques. These techniques are useful for understanding and debugging constraint systems in visual mapping rules. Using these techniques, we developed a browser for constraints in VSR of TRIP systems. It is implemented using Java and integrated with the animation viewer of TRIP2a/3D.報告番号: 乙15762 ; 学位授与年月日: 2003-09-16 ; 学位の種別: 論文博士 ; 学位の種類: 博士(理学) ; 学位記番号: 第15762号 ; 研究科・専攻: 理学系研究