A Joint Audio-Visual Approach to Audio Localization

Localization of audio sources is an important research problem, e.g.,to facilitate noise reduction. In the recent years, the problem hasbeen tackled using distributed microphone arrays (DMA). A commonapproach is to apply direction-of-arrival (DOA) estimation oneach array (denoted as nodes), and then map the DOA estimates toa location. In practice, however, the individual nodes contain fewmicrophones, limiting the DOA estimation accuracy and, thereby,also the localization performance. We investigate a new approach,where range estimates are also obtained and utilized from each node,e.g., using time-of-flight cameras. Moreover, we propose an optimalmethod for weighting such DOA and range information for audiolocalization. Our experiments on both synthetic and real data showthat there is a clear, potential advantage of using the joint audiovisuallocalization framework.Localization of audio sources is an important research problem, e.g., to facilitate noise reduction. In the recent years, the problem has been tackled using distributed microphone arrays (DMA). A common approach is to apply direction-of-arrival (DOA) estimation on each array (denoted as nodes), and then map the DOA estimates to a location. In practice, however, the individual nodes contain few microphones, limiting the DOA estimation accuracy and, thereby, also the localization performance. We investigate a new approach, where range estimates are also obtained and utilized from each node, e.g., using time-of-flight cameras. Moreover, we propose an optimal method for weighting such DOA and range information for audio localization. Our experiments on both synthetic and real data show that there is a clear, potential advantage of using the joint audiovisual localization framework