Towards Intelligibility-Oriented Audio-Visual Speech Enhancement
  Existing deep learning (DL) based speech enhancement approaches are generally
optimised to minimise the distance between clean and enhanced speech features.
These often result in improved speech quality however they suffer from a lack
of generalisation and may not deliver the required speech intelligibility in
real noisy situations. In an attempt to address these challenges, researchers
have explored intelligibility-oriented (I-O) loss functions and integration of
audio-visual (AV) information for more robust speech enhancement (SE). In this
paper, we introduce DL based I-O SE algorithms exploiting AV information, which
is a novel and previously unexplored research direction. Specifically, we
present a fully convolutional AV SE model that uses a modified short-time
objective intelligibility (STOI) metric as a training cost function. To the
best of our knowledge, this is the first work that exploits the integration of
AV modalities with an I-O based loss function for SE. Comparative experimental
results demonstrate that our proposed I-O AV SE framework outperforms
audio-only (AO) and AV models trained with conventional distance-based loss
functions, in terms of standard objective evaluation measures when dealing with
unseen speakers and noises.
