Seeing (movement) is believing: the effect of motion on perception of automatic systems performance

In this article, we report on one lab study and seven follow-up studies on a crowdsourcing platform designed to investigate the potential of animation cues to influence users’ perception of two smart systems: a handwriting recognition and a part-of-speech tagging system. Results from the first three studies indicate that animation cues can influence a participant’s perception of both systems’ performance. The subsequent three studies, designed to try and identify an explanation for this effect, suggest that this effect is related to the participants’ mental model of the smart system. The last two studies were designed to characterize the effect more in detail, and they revealed that different amounts of animation do not seem to create substantial differences and that the effect persists even when the system’s performance decreases, but only when the difference in performance level between the systems being compared is small.