SOFT NONNEGATIVE MATRIX CO-FACTORIZATION WITH APPLICATION TO MULTIMODAL SPEAKER DIARIZATION

This paper presents a new method for bimodal nonnegative matrix factorization (NMF). This method is well-suited to sit-uations where two streams of data are concurrently analyzed and are expected to be related by loosely common factors. It allows for a soft co-factorization, which takes into account the relationship that exists between the modalities being pro-cessed, but returns different factors for distinct modalities. There is no need that the data related with each modality live in the same feature space; there is also no need that they have the same dimensionality. The co-factorization is obtained via a majorization-minimization (MM) algorithm. The behavior of the method is illustrated on both synthetic and real-world data. In particular, we show that exploiting the correlation be-tween audio and video modalities in edited talk-show videos improve speaker diarization results. Index Terms—Nonnegative matrix factorization, co-factorization, multimodality, speaker diarization 1