Speech enhancement aided end-to-end multi-task learning for voice
  activity detection
  Robust voice activity detection (VAD) is a challenging task in low
signal-to-noise (SNR) environments. Recent studies show that speech enhancement
is helpful to VAD, but the performance improvement is limited. To address this
issue, here we propose a speech enhancement aided end-to-end multi-task model
for VAD. The model has two decoders, one for speech enhancement and the other
for VAD. The two decoders share the same encoder and speech separation network.
Unlike the direct thought that takes two separated objectives for VAD and
speech enhancement respectively, here we propose a new joint optimization
objective -- VAD-masked scale-invariant source-to-distortion ratio (mSI-SDR).
mSI-SDR uses VAD information to mask the output of the speech enhancement
decoder in the training process. It makes the VAD and speech enhancement tasks
jointly optimized not only at the shared encoder and separation network, but
also at the objective level. It also satisfies real-time working requirement
theoretically. Experimental results show that the multi-task method
significantly outperforms its single-task VAD counterpart. Moreover, mSI-SDR
outperforms SI-SDR in the same multi-task setting.
