Evaluating Conversation Agent Impact on Student Experience in a Distance Education Course

We explore the efficacy of conversation agents operating as an instructional aid in a distance education course. Two aspects of efficacy are considered—conversation agent impact on student perceptions of the experience, and how different design features of the agent affect student perceptions of engagement. Evaluation of the agent is accomplished by collecting data from 24 undergraduate participants separated into random groups. We conduct two rounds of mixedmethod evaluation. Between the two rounds, a modification to the agent occurs based on the outcome of the first evaluation. Findings include limitations related to phrasing and data persistence features of the design that initially yielded less-than-favorable perceptions of the agent prototype