Best of Both Worlds: Multi-task Audio-Visual Automatic Speech Recognition and Active Speaker Detection
Under noisy conditions, automatic speech recognition (ASR) can greatly
benefit from the addition of visual signals coming from a video of the
speaker's face. However, when multiple candidate speakers are visible this
traditionally requires solving a separate problem, namely active speaker
detection (ASD), which entails selecting at each moment in time which of the
visible faces corresponds to the audio. Recent work has shown that we can solve
both problems simultaneously by employing an attention mechanism over the
competing video tracks of the speakers' faces, at the cost of sacrificing some
accuracy on active speaker detection. This work closes this gap in active
speaker detection accuracy by presenting a single model that can be jointly
trained with a multi-task loss. By combining the two tasks during training we
reduce the ASD classification accuracy by approximately 25%, while
simultaneously improving the ASR performance when compared to the multi-person
baseline trained exclusively for ASR.