CochleaNet: A Robust Language-independent Audio-Visual Model for Speech
  Enhancement
  Noisy situations cause huge problems for suffers of hearing loss as hearing
aids often make the signal more audible but do not always restore the
intelligibility. In noisy settings, humans routinely exploit the audio-visual
(AV) nature of the speech to selectively suppress the background noise and to
focus on the target speaker. In this paper, we present a causal, language,
noise and speaker independent AV deep neural network (DNN) architecture for
speech enhancement (SE). The model exploits the noisy acoustic cues and noise
robust visual cues to focus on the desired speaker and improve the speech
intelligibility. To evaluate the proposed SE framework a first of its kind AV
binaural speech corpus, called ASPIRE, is recorded in real noisy environments
including cafeteria and restaurant. We demonstrate superior performance of our
approach in terms of objective measures and subjective listening tests over the
state-of-the-art SE approaches as well as recent DNN based SE models. In
addition, our work challenges a popular belief that a scarcity of
multi-language large vocabulary AV corpus and wide variety of noises is a major
bottleneck to build a robust language, speaker and noise independent SE
systems. We show that a model trained on synthetic mixture of Grid corpus (with
33 speakers and a small English vocabulary) and ChiME 3 Noises (consisting of
only bus, pedestrian, cafeteria, and street noises) generalise well not only on
large vocabulary corpora but also on completely unrelated languages (such as
Mandarin), wide variety of speakers and noises.
