Robust Unsupervised Audio-visual Speech Enhancement Using a Mixture of
  Variational Autoencoders
  Recently, an audio-visual speech generative model based on variational
autoencoder (VAE) has been proposed, which is combined with a nonnegative
matrix factorization (NMF) model for noise variance to perform unsupervised
speech enhancement. When visual data is clean, speech enhancement with
audio-visual VAE shows a better performance than with audio-only VAE, which is
trained on audio-only data. However, audio-visual VAE is not robust against
noisy visual data, e.g., when for some video frames, speaker face is not
frontal or lips region is occluded. In this paper, we propose a robust
unsupervised audio-visual speech enhancement method based on a per-frame VAE
mixture model. This mixture model consists of a trained audio-only VAE and a
trained audio-visual VAE. The motivation is to skip noisy visual frames by
switching to the audio-only VAE model. We present a variational
expectation-maximization method to estimate the parameters of the model.
Experiments show the promising performance of the proposed method.
