Environmental sound recognition with time-frequency audio

Abstract—The paper considers the task of recognizing envi-ronmental sounds for the understanding of a scene or context surrounding an audio sensor. A variety of features have been pro-posed for audio recognition, including the popular Mel-frequency cepstral coefficients (MFCCs) which describe the audio spectral shape. Environmental sounds, such as chirpings of insects and sounds of rain which are typically noise-like with a broad flat spec-trum, may include strong temporal domain signatures. However, only few temporal-domain features have been developed to char-acterize such diverse audio signals previously. Here, we perform an empirical feature analysis for audio environment characteriza-tion and propose to use the matching pursuit (MP) algorithm to obtain effective time–frequency features. The MP-based method utilizes a dictionary of atoms for feature selection, resulting in a flexible, intuitive and physically interpretable set of features. The MP-based feature is adopted to supplement the MFCC features to yield higher recognition accuracy for environmental sounds. Extensive experiments are conducted to demonstrate the effec-tiveness of these joint features for unstructured environmental sound classification, including listening tests to study human recognition capabilities. Our recognition system has shown to produce comparable performance as human listeners. Index Terms—Audio classification, auditory scene recognition, data representation, feature extraction, feature selection, matching pursuit, Mel-frequency cepstral coefficient (MFCC). I