Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot
  Learning with Knowledge Distillation
  In realistic speech enhancement settings for end-user devices, we often
encounter only a few speakers and noise types that tend to reoccur in the
specific acoustic environment. We propose a novel personalized speech
enhancement method to adapt a compact denoising model to the test-time
specificity. Our goal in this test-time adaptation is to utilize no clean
speech target of the test speaker, thus fulfilling the requirement for
zero-shot learning. To complement the lack of clean utterance, we employ the
knowledge distillation framework. Instead of the missing clean utterance
target, we distill the more advanced denoising results from an overly large
teacher model, and use it as the pseudo target to train the small student
model. This zero-shot learning procedure circumvents the process of collecting
users' clean speech, a process that users are reluctant to comply due to
privacy concerns and technical difficulty of recording clean voice. Experiments
on various test-time conditions show that the proposed personalization method
achieves significant performance gains compared to larger baseline networks
trained from a large speaker- and noise-agnostic datasets. In addition, since
the compact personalized models can outperform larger general-purpose models,
we claim that the proposed method performs model compression with no loss of
denoising performance.
