A Study of Designing Compact Audio-Visual Wake Word Spotting System
  Based on Iterative Fine-Tuning in Neural Network Pruning
  Audio-only-based wake word spotting (WWS) is challenging under noisy
conditions due to environmental interference in signal transmission. In this
paper, we investigate on designing a compact audio-visual WWS system by
utilizing visual information to alleviate the degradation. Specifically, in
order to use visual information, we first encode the detected lips to
fixed-size vectors with MobileNet and concatenate them with acoustic features
followed by the fusion network for WWS. However, the audio-visual model based
on neural networks requires a large footprint and a high computational
complexity. To meet the application requirements, we introduce a neural network
pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning
manner (LTH-IF), to the single-modal and multi-modal models, respectively.
Tested on our in-house corpus for audio-visual WWS in a home TV scene, the
proposed audio-visual system achieves significant performance improvements over
the single-modality (audio-only or video-only) system under different noisy
conditions. Moreover, LTH-IF pruning can largely reduce the network parameters
and computations with no degradation of WWS performance, leading to a potential
product solution for the TV wake-up scenario.
