Correlating Subword Articulation with Lip Shapes for Embedding Aware
  Audio-Visual Speech Enhancement
  In this paper, we propose a visual embedding approach to improving embedding
aware speech enhancement (EASE) by synchronizing visual lip frames at the phone
and place of articulation levels. We first extract visual embedding from lip
frames using a pre-trained phone or articulation place recognizer for
visual-only EASE (VEASE). Next, we extract audio-visual embedding from noisy
speech and lip videos in an information intersection manner, utilizing a
complementarity of audio and visual features for multi-modal EASE (MEASE).
Experiments on the TCD-TIMIT corpus corrupted by simulated additive noises show
that our proposed subword based VEASE approach is more effective than
conventional embedding at the word level. Moreover, visual embedding at the
articulation place level, leveraging upon a high correlation between place of
articulation and lip shapes, shows an even better performance than that at the
phone level. Finally the proposed MEASE framework, incorporating both audio and
visual embedding, yields significantly better speech quality and
intelligibility than those obtained with the best visual-only and audio-only
EASE systems.
