Audio Captioning with Composition of Acoustic and Semantic Information
  Generating audio captions is a new research area that combines audio and
natural language processing to create meaningful textual descriptions for audio
clips. To address this problem, previous studies mostly use the encoder-decoder
based models without considering semantic information. To fill this gap, we
present a novel encoder-decoder architecture using bi-directional Gated
Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic
embedding by obtaining subjects and verbs from the audio clip captions and
combine these embedding with audio embedding to feed the BiGRU-based
encoder-decoder model. To enable semantic embeddings for the test audios, we
introduce a Multilayer Perceptron classifier to predict the semantic embeddings
of those clips. We also present exhaustive experiments to show the efficiency
of different features and datasets for our proposed model the audio captioning
task. To extract audio features, we use the log Mel energy features, VGGish
embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive
experiments on two audio captioning datasets Clotho and AudioCaps show that our
proposed model outperforms state-of-the-art audio captioning models across
different evaluation metrics and using the semantic information improves the
captioning performance. Keywords: Audio captioning; PANNs; VGGish; GRU; BiGRU.
