Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss
  We present an audio-visual speech separation learning method that considers
the correspondence between the separated signals and the visual signals to
reflect the speech characteristics during training. Audio-visual speech
separation is a technique to estimate the individual speech signals from a
mixture using the visual signals of the speakers. Conventional studies on
audio-visual speech separation mainly train the separation model on the
audio-only loss, which reflects the distance between the source signals and the
separated signals. However, conventional losses do not reflect the
characteristics of the speech signals, including the speaker's characteristics
and phonetic information, which leads to distortion or remaining noise. To
address this problem, we propose the cross-modal correspondence (CMC) loss,
which is based on the cooccurrence of the speech signal and the visual signal.
Since the visual signal is not affected by background noise and contains
speaker and phonetic information, using the CMC loss enables the audio-visual
speech separation model to remove noise while preserving the speech
characteristics. Experimental results demonstrate that the proposed method
learns the cooccurrence on the basis of CMC loss, which improves separation
performance.
