Empirical studies on a tangible user interface for technology-based assessment: Insights and emerging challenges

The assessment of higher order thinking skills should measure knowledge and procedure as well as attitudes and dispositions. It can be considered as multidimensional regarding the cognitive processes involved in solving, for instance, a complex problem. This paper reports on two empirical studies in which a so called tangible user interface (TUI) was used for the assessment. A simple matching item, suitable for measuring recall of factual knowledge, as well as a simulation item with the potential to assess higher order thinking skills are presented. The existing knowledge about using such systems for assessment is limited. We therefore focused on evaluation of the usability, specifically the user experience (UX) and interviewed experts in order to derive a list of tensions and also advantages. A first study showed that the available constructs for evaluating usability and UX need to be adapted to reflect the collaborative problem solving setting. The second study revealed that the majority of participants rated the system positively from the usability and UX perspective. Some tensions mentioned by the experts were related to the first phase of getting used to the system: When should the system start to assess the observed solving strategies? How could the system identify single atomic contributions (i.e. single experiments with the simulation parameters)? How could we include activities outside the interactive surface of the table? The table supports the user to ‘recognise the perspective of others', an important sub-skill of collaborative problem solving - how can the table track these activities? Based on the outcomes of the studies we identified eight topics. We discuss the related tensions for each and the advantages between the new technology and technology-based assessment which will impact the future development of using TUI for assessment as well as the design of assessment models and methods