Personal VAD 2.0: Optimizing Personal Voice Activity Detection for
  On-Device Speech Recognition
  Personalization of on-device speech recognition (ASR) has seen explosive
growth in recent years, largely due to the increasing popularity of personal
assistant features on mobile devices and smart home speakers. In this work, we
present Personal VAD 2.0, a personalized voice activity detector that detects
the voice activity of a target speaker, as part of a streaming on-device ASR
system. Although previous proof-of-concept studies have validated the
effectiveness of Personal VAD, there are still several critical challenges to
address before this model can be used in production: first, the quality must be
satisfactory in both enrollment and enrollment-less scenarios; second, it
should operate in a streaming fashion; and finally, the model size should be
small enough to fit a limited latency and CPU/Memory budget. To meet the
multi-faceted requirements, we propose a series of novel designs: 1) advanced
speaker embedding modulation methods; 2) a new training paradigm to generalize
to enrollment-less conditions; 3) architecture and runtime optimizations for
latency and resource restrictions. Extensive experiments on a realistic speech
recognition system demonstrated the state-of-the-art performance of our
proposed method.
