MusCaps: Generating Captions for Music Audio
  Content-based music information retrieval has seen rapid progress with the
adoption of deep learning. Current approaches to high-level music description
typically make use of classification models, such as in auto-tagging or genre
and mood classification. In this work, we propose to address music description
via audio captioning, defined as the task of generating a natural language
description of music audio content in a human-like manner. To this end, we
present the first music audio captioning model, MusCaps, consisting of an
encoder-decoder with temporal attention. Our method combines convolutional and
recurrent neural network architectures to jointly process audio-text inputs
through a multimodal encoder and leverages pre-training on audio data to obtain
representations that effectively capture and summarise musical features in the
input. Evaluation of the generated captions through automatic metrics shows
that our method outperforms a baseline designed for non-music audio captioning.
Through an ablation study, we unveil that this performance boost can be mainly
attributed to pre-training of the audio encoder, while other design choices -
modality fusion, decoding strategy and the use of attention - contribute only
marginally. Our model represents a shift away from classification-based music
description and combines tasks requiring both auditory and linguistic
understanding to bridge the semantic gap in music information retrieval.
