Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual
  Speech Recognition
  One crucial challenge of real-world multilingual speech recognition is the
long-tailed distribution problem, where some resource-rich languages like
English have abundant training data, but a long tail of low-resource languages
have varying amounts of limited training data. To overcome the long-tail
problem, in this paper, we propose Adapt-and-Adjust (A2), a transformer-based
multi-task learning framework for end-to-end multilingual speech recognition.
The A2 framework overcomes the long-tail problem via three techniques: (1)
exploiting a pretrained multilingual language model (mBERT) to improve the
performance of low-resource languages; (2) proposing dual adapters consisting
of both language-specific and language-agnostic adaptation with minimal
additional parameters; and (3) overcoming the class imbalance, either by
imposing class priors in the loss during training or adjusting the logits of
the softmax output during inference. Extensive experiments on the CommonVoice
corpus show that A2 significantly outperforms conventional approaches.
