Speaker Diarization using Eye-gaze Information in Multi-party Conversations

We present a novel speaker diarization method by using eye-gaze information in multi-party conversations. In real environ-ments, speaker diarization or speech activity detection of each participant of the conversation is challenging because of distant talking and ambient noise. In contrast, eye-gaze information is robust against acoustic degradation, and it is presumed that eye-gaze behavior plays an important role in turn-taking and thus in predicting utterances. The proposed method stochastically integrates eye-gaze information with acoustic information for speaker diarization. Specifically, three models are investigated for multi-modal integration in this paper. Experimental eval-uations in real poster sessions demonstrate that the proposed method improves accuracy of speaker diarization from the base-line acoustic method. Index Terms: speaker diarization, multi-modal interaction, eye-gaz