SALSA: spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection

Sound event localization and detection (SELD) consists of two subtasks, which are sound event detection and direction-of-arrival estimation. While sound event detection mainly relies on time-frequency patterns to distinguish different sound classes, direction-of-arrival estimation uses amplitude and/or phase differences between microphones to estimate source directions. As a result, it is often difficult to jointly optimize these two subtasks. We propose a novel feature called Spatial cue-Augmented Log-SpectrogrAm (SALSA) with exact time-frequency mapping between the signal power and the source directional cues, which is crucial for resolving overlapping sound sources. The SALSA feature consists of multichannel log-spectrograms stacked along with the normalized principal eigenvector of the spatial covariance matrix at each corresponding time-frequency bin. Depending on the microphone array format, the principal eigenvector can be normalized differently to extract amplitude and/or phase differences between the microphones. As a result, SALSA features are applicable for different microphone array formats such as first-order ambisonics (FOA) and multichannel microphone array (MIC). Experimental results on the TAUNIGENS Spatial Sound Events 2021 dataset with directional interferences showed that SALSA features outperformed other state-of-the-art features. Specifically, the use of SALSA features in the FOA format increased the F1 score and localization recall by 6 % each, compared to the multichannel log-mel spectrograms with intensity vectors. For the MIC format, using SALSA features increased F1 score and localization recall by 16 % and 7 %, respectively, compared to using multichannel logmel spectrograms with generalized cross-correlation spectra.Ministry of Education (MOE)Nanyang Technological UniversitySubmitted/Accepted versionThis work was supported in part by the SingaporeMinistry of Education Academic Research Fund Tier-2, under Research Grant MOE2017- T2-2-060, and in part by Google Cloud Research Credits Program under Award GCP205559654. K. N. Watcharasupat further acknowledges the support from the CN Yang Scholars Programme, Nanyang Technological University, Singapore