Cross-Modal learning for Audio-Visual Video Parsing
  In this paper, we present a novel approach to the audio-visual video parsing
(AVVP) task that demarcates events from a video separately for audio and visual
modalities. The proposed parsing approach simultaneously detects the temporal
boundaries in terms of start and end times of such events. We show how AVVP can
benefit from the following techniques geared towards effective cross-modal
learning: (i) adversarial training and skip connections (ii) global context
aware attention and, (iii) self-supervised pretraining using an audio-video
grounding objective to obtain cross-modal audio-video representations. We
present extensive experimental evaluations on the Look, Listen, and Parse (LLP)
dataset and show that we outperform the state-of-the-art Hybrid Attention
Network (HAN) on all five metrics proposed for AVVP. We also present several
ablations to validate the effect of pretraining, global attention and
adversarial training.
