Audiovisual transfer learning for audio tagging and sound event
  detection
  We study the merit of transfer learning for two sound recognition problems,
i.e., audio tagging and sound event detection. Employing feature fusion, we
adapt a baseline system utilizing only spectral acoustic inputs to also make
use of pretrained auditory and visual features, extracted from networks built
for different tasks and trained with external data. We perform experiments with
these modified models on an audiovisual multi-label data set, of which the
training partition contains a large number of unlabeled samples and a smaller
amount of clips with weak annotations, indicating the clip-level presence of 10
sound categories without specifying the temporal boundaries of the active
auditory events. For clip-based audio tagging, this transfer learning method
grants marked improvements. Addition of the visual modality on top of audio
also proves to be advantageous in this context. When it comes to generating
transcriptions of audio recordings, the benefit of pretrained features depends
on the requested temporal resolution: for coarse-grained sound event detection,
their utility remains notable. But when more fine-grained predictions are
required, performance gains are strongly reduced due to a mismatch between the
problem at hand and the goals of the models from which the pretrained vectors
were obtained.
