Towards gesture-based literacy training with a virtual agent

Bergmann K. Towards gesture-based literacy training with a virtual agent. In: Jokinen K, Vels M, eds. Proceedings of the 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping Electronic Conference Proceedings; 2015: 113-121.Illiteracy remains a persistent problem all over the world. Especially people with cognitive impairments are affected and, at the same time, often excluded from interventions because courses and e-learning materials are not tailored to the particular needs of these people. A teaching method that is occasionally applied in special schools is a gesture-based training: Learning of letter-sound pairs is supported by associated gestures. In this paper first steps towards realizing this methodology with a virtual teacher are presented. We apply a user-centered approach and present a first prototype system which is used to set up an evaluation with learners from the target audience. In the study, participants learned with the system in consecutive learning sessions over three days. We present results with regard to the general acceptability of the system, problems the learners had as well as issues of the system they liked. The outcome of the study serves to inform future versions of the prototype system