Crossmodal Matching of Speakers using Lip and Voice Features in Temporally Non-overlapping Audio and Video Streams

Person identification using audio (speech) and vi-sual (facial appearance, static or dynamic) modalities, either independently or jointly, is a thoroughly investi-gated problem in pattern recognition. In this work, we explore a novel task: person identification in a cross-modal scenario, i.e., matching the speaker in an au-dio recording to the same speaker in a video recording, where the two recordings have been made during differ-ent sessions, using speaker specific information which is common to both the audio and video modalities. Several recent psychological studies have shown how humans can indeed perform this task with an accuracy signifi-cantly higher than chance. Here we propose two sys-tems which can solve this task comparably well, using purely pattern recognition techniques. We hypothesize that such systems could be put to practical use in multi-modal biometric and surveillance systems. 1