Audio-visual synchronization recovery in multimedia content

This paper proposes a method recovering audio-visual syn-chronization of multimedia content. It exploits the correlation between the acoustic and the visual signals in order to esti-mate the audio-visual drift existing in the content. By shifting the audio signal relative to the visual signal, the estimation of the drift is obtained by searching for the shift producing the maximal audio-visual correlation. We consider two cor-relation measures, namely, mutual information and canonical correlation, and compare their performance. Experimental re-sults demonstrate that the method using the canonical corre-lation is effective in recovering the audio-visual synchroniza-tion for both speech and non-speech sequences. Index Terms — Audio-visual synchronization, mutual in-formation, canonical correlation, multimedia 1