On Attention Modules for Audio-Visual Synchronization
  With the development of media and networking technologies, multimedia
applications ranging from feature presentation in a cinema setting to video on
demand to interactive video conferencing are in great demand. Good
synchronization between audio and video modalities is a key factor towards
defining the quality of a multimedia presentation. The audio and visual signals
of a multimedia presentation are commonly managed by independent workflows -
they are often separately authored, processed, stored and even delivered to the
playback system. This opens up the possibility of temporal misalignment between
the two modalities - such a tendency is often more pronounced in the case of
produced content (such as movies).
  To judge whether audio and video signals of a multimedia presentation are
synchronized, we as humans often pay close attention to discriminative
spatio-temporal blocks of the video (e.g. synchronizing the lip movement with
the utterance of words, or the sound of a bouncing ball at the moment it hits
the ground). At the same time, we ignore large portions of the video in which
no discriminative sounds exist (e.g. background music playing in a movie).
Inspired by this observation, we study leveraging attention modules for
automatically detecting audio-visual synchronization. We propose neural network
based attention modules, capable of weighting different portions
(spatio-temporal blocks) of the video based on their respective discriminative
power. Our experiments indicate that incorporating attention modules yields
state-of-the-art results for the audio-visual synchronization classification
problem.
