Towards Robust Real-time Audio-Visual Speech Enhancement
  The human brain contextually exploits heterogeneous sensory information to
efficiently perform cognitive tasks including vision and hearing. For example,
during the cocktail party situation, the human auditory cortex contextually
integrates audio-visual (AV) cues in order to better perceive speech. Recent
studies have shown that AV speech enhancement (SE) models can significantly
improve speech quality and intelligibility in very low signal to noise ratio
(SNR) environments as compared to audio-only SE models. However, despite
significant research in the area of AV SE, development of real-time processing
models with low latency remains a formidable technical challenge. In this
paper, we present a novel framework for low latency speaker-independent AV SE
that can generalise on a range of visual and acoustic noises. In particular, a
generative adversarial networks (GAN) is proposed to address the practical
issue of visual imperfections in AV SE. In addition, we propose a deep neural
network based real-time AV SE model that takes into account the cleaned visual
speech output from GAN to deliver more robust SE. The proposed framework is
evaluated on synthetic and real noisy AV corpora using objective speech quality
and intelligibility metrics and subjective listing tests. Comparative
simulation results show that our real time AV SE framework outperforms
state-of-the-art SE approaches, including recent DNN based SE models.
