AV Taris: Online Audio-Visual Speech Recognition
  In recent years, Automatic Speech Recognition (ASR) technology has approached
human-level performance on conversational speech under relatively clean
listening conditions. In more demanding situations involving distant
microphones, overlapped speech, background noise, or natural dialogue
structures, the ASR error rate is at least an order of magnitude higher. The
visual modality of speech carries the potential to partially overcome these
challenges and contribute to the sub-tasks of speaker diarisation, voice
activity detection, and the recovery of the place of articulation, and can
compensate for up to 15dB of noise on average. This article develops AV Taris,
a fully differentiable neural network model capable of decoding audio-visual
speech in real time. We achieve this by connecting two recently proposed models
for audio-visual speech integration and online speech recognition, namely AV
Align and Taris. We evaluate AV Taris under the same conditions as AV Align and
Taris on one of the largest publicly available audio-visual speech datasets,
LRS2. Our results show that AV Taris is superior to the audio-only variant of
Taris, demonstrating the utility of the visual modality to speech recognition
within the real time decoding framework defined by Taris. Compared to an
equivalent Transformer-based AV Align model that takes advantage of full
sentences without meeting the real-time requirement, we report an absolute
degradation of approximately 3% with AV Taris. As opposed to the more popular
alternative for online speech recognition, namely the RNN Transducer, Taris
offers a greatly simplified fully differentiable training pipeline. As a
consequence, AV Taris has the potential to popularise the adoption of
Audio-Visual Speech Recognition (AVSR) technology and overcome the inherent
limitations of the audio modality in less optimal listening conditions.
