An empirical study of the effect of agent competence on user performance and perception

We studied the role of the competence of a user interface agent/assistant that helped users to learn and use a new text editor. Participants in the study made a set of prescribed changes to a document via the editor with the aid of one of four interface agents. Participants could ask questions out loud to the agent and the agent would respond using a synthesized voice; the agent would also make proactive suggestions. The agents varied in the quality of responses and suggestions made. One group of participants were provided with a help screen as well as the agent. We focused on assessing the relation between users ' objective performance, interaction style, and subjective experience. Results revealed that the perceived utility of the agent was influenced by the types of errors made by the agent, while participants ' subjective impressions of the agent related to the perceptions of its representation. In addition, allowing participants to choose their preferred assistance style(s) (agent vs. online-help) improved objective performance. We correlate quantitative findings with qualitative interview data and discuss implications for the design and the implementation of systems with interface agents