Listenership in human-agent collectives: a study of unidirectional instruction-giving

Research in nonverbal listenership behaviour and instruction-giving has focused on interaction with people while paying inadequate attention to human-agent interaction even as recent research indicates that, increasing pervasive computing is significantly changing how humans interact with intelligent software agents and extending the boundaries of discourse to contexts including satellite navigation systems giving directions to drivers, self-checkout machines in supermarkets and intelligent personal assistants on smartphones. This thesis reports studies that use spontaneous listener facial actions and gestures to understand the nature and pattern of spontaneous nonverbal listenership behaviours, identification and communication in instruction-giving contexts. The research methodology used is as follows. Participants who are all L1 speakers of English (forty-eight in Study 1, six in Study 2) were tasked with assembling two Lego models using vague verbal instructions from a computer interface in Study 1 and a human instructor in Study 2 with a 15-minute time limit per iteration. The interface in study1uses three voices of which two are synthesised and one is non-synthesised human recording by a voice actor while Study 2 used a live human voice. A 24-hour long multimodal corpus was built and analysed from interactions between participants and the interface in Study 1 while a 3-hour multimodal corpus was developed from Study 2. The multimodal corpus was annotated for marked facial actions and gestures occurring at points when participants requested that instructors repeat instructions. Participant requests were nonverbal in HAI and a combination of nonverbal and verbal instructions in HHI contexts. The repetitions were quantified and classified into nine typologies. The results reveal key findings regarding the use of spontaneous nonverbal listenership behaviours as pragmatic markers indicating listener comprehension or incomprehension of instructions, perception of instructor-identities, projection of attitudes, meaning-development, task-execution strategies and interaction management even though, the agent could not attend to them in the same way a human can. Using these results, the thesis submits that there are potentials for applied linguistics theories and research to be used to identify and understand pathways to make agents more responsive to human behaviour, make human-agent interaction more credible and provides a theoretical foundation for future multidisciplinary research