On the Impact and Interplay of Input Representations and Network Architectures for Automatic Music Tagging

Automatic music tagging systems have once more gained relevance over the last years, not least through their use in applications such as music recommender systems. State-of-the-art systems are based on a variant of convolutional neural networks (CNNs) and use some type of time-frequency audio representation as input, in a fitting combination to predict semantic tags available through expert or crowd-based annotation. In this work we systematically compare five widely used audio input representations (STFT, CQT, Mel spectrograms, MFCCs, and raw audio waveform) using five established convolutional neural network architectures (MusicCNN, VGG16, ResNet, a Squeeze and Excitation Network (SeNet), as well as a newly proposed MusicCNN variant using dilated convolutions) for the task of music tag prediction. Performance of all factor combinations are measured on two distinct tagging datasets, namely MagnaTagATune and MTG Jamendo. A two-way ANOVA shows that both input representation and model architecture significantly impact the classification results. Despite differently sized input representations and practical impact on model training, we find that using STFT as input representations provides the best results overall and on specific tag categories (genre, instrument, mood), while other representations show less consistent behavior in these regards. Furthermore, the proposed dilated convolutional architecture shows significant performance improvements for all input representations except raw waveform