Multimodal Attention Fusion for Target Speaker Extraction
  Target speaker extraction, which aims at extracting a target speaker's voice
from a mixture of voices using audio, visual or locational clues, has received
much interest. Recently an audio-visual target speaker extraction has been
proposed that extracts target speech by using complementary audio and visual
clues. Although audio-visual target speaker extraction offers a more stable
performance than single modality methods for simulated data, its adaptation
towards realistic situations has not been fully explored as well as evaluations
on real recorded mixtures. One of the major issues to handle realistic
situations is how to make the system robust to clue corruption because in real
recordings both clues may not be equally reliable, e.g. visual clues may be
affected by occlusions. In this work, we propose a novel attention mechanism
for multi-modal fusion and its training methods that enable to effectively
capture the reliability of the clues and weight the more reliable ones. Our
proposals improve signal to distortion ratio (SDR) by 1.0 dB over conventional
fusion mechanisms on simulated data. Moreover, we also record an audio-visual
dataset of simultaneous speech with realistic visual clue corruption and show
that audio-visual target speaker extraction with our proposals successfully
work on real data.
