How to Teach DNNs to Pay Attention to the Visual Modality in Speech
  Recognition
  Audio-Visual Speech Recognition (AVSR) seeks to model, and thereby exploit,
the dynamic relationship between a human voice and the corresponding mouth
movements. A recently proposed multimodal fusion strategy, AV Align, based on
state-of-the-art sequence to sequence neural networks, attempts to model this
relationship by explicitly aligning the acoustic and visual representations of
speech. This study investigates the inner workings of AV Align and visualises
the audio-visual alignment patterns. Our experiments are performed on two of
the largest publicly available AVSR datasets, TCD-TIMIT and LRS2. We find that
AV Align learns to align acoustic and visual representations of speech at the
frame level on TCD-TIMIT in a generally monotonic pattern. We also determine
the cause of initially seeing no improvement over audio-only speech recognition
on the more challenging LRS2. We propose a regularisation method which involves
predicting lip-related Action Units from visual representations. Our
regularisation method leads to better exploitation of the visual modality, with
performance improvements between 7% and 30% depending on the noise level.
Furthermore, we show that the alternative Watch, Listen, Attend, and Spell
network is affected by the same problem as AV Align, and that our proposed
approach can effectively help it learn visual representations. Our findings
validate the suitability of the regularisation method to AVSR and encourage
researchers to rethink the multimodal convergence problem when having one
dominant modality.
