A Real-Time Architecture for Embodied Conversational Agents: Beyond Turn-Taking

Abstract—We describe the design, implementation and use of a middleware system, called DiscoRT, to support the development of virtual and robotic conversational agents. The use cases for this system include handling conversational and event-based interrup-tions, and supporting engagement maintenance behaviors, such as turn-taking, backchanneling, directed gaze and face tracking. The multi-threaded architecture of the system includes both “hard” and “soft ” real-time scheduling and integrates with an existing collaborative discourse manager, called Disco. We have used the system to build a substantial conversational agent that is about to undergo long-term field studies. Keywords-engagement; interruption; turn-taking; backchannel; barge-in; schema; BML; arbitration. I