Improved Lite Audio-Visual Speech Enhancement
  Numerous studies have investigated the effectiveness of audio-visual
multimodal learning for speech enhancement (AVSE) tasks, seeking a solution
that uses visual data as auxiliary and complementary input to reduce the noise
of noisy speech signals. Recently, we proposed a lite audio-visual speech
enhancement (LAVSE) algorithm for a car-driving scenario. Compared to
conventional AVSE systems, LAVSE requires less online computation and to some
extent solves the user privacy problem on facial data. In this study, we extend
LAVSE to improve its ability to address three practical issues often
encountered in implementing AVSE systems, namely, the additional cost of
processing visual data, audio-visual asynchronization, and low-quality visual
data. The proposed system is termed improved LAVSE (iLAVSE), which uses a
convolutional recurrent neural network architecture as the core AVSE model. We
evaluate iLAVSE on the Taiwan Mandarin speech with video dataset. Experimental
results confirm that compared to conventional AVSE systems, iLAVSE can
effectively overcome the aforementioned three practical issues and can improve
enhancement performance. The results also confirm that iLAVSE is suitable for
real-world scenarios, where high-quality audio-visual sensors may not always be
available.
