Domain Prompts: Towards memory and compute efficient domain adaptation
  of ASR systems
  Automatic Speech Recognition (ASR) systems have found their use in numerous
industrial applications in very diverse domains creating a need to adapt to new
domains with small memory and deployment overhead. In this work, we introduce
domain prompts, a methodology that involves training a small number of domain
embedding parameters to prime a Transformer-based Language Model (LM) to a
particular domain. Using this domain-adapted LM for rescoring ASR hypotheses
can achieve 7-13% WER reduction for a new domain with just 1000 unlabeled
textual domain-specific sentences and a handful of additional parameters. Our
method can match or even beat the performance of models fully fine-tuned
towards a particular domain with only 0.02% of the parameters. Given the
parameter efficiency and negligible deployment overhead, our experiments
showcase that such a method is an ideal choice for on-the-fly adaptation of LMs
used in ASR systems to progressively scale it to new domains.
