Adversarial Training for Multilingual Acoustic Modeling
  Multilingual training has been shown to improve acoustic modeling performance
by sharing and transferring knowledge in modeling different languages.
Knowledge sharing is usually achieved by using common lower-level layers for
different languages in a deep neural network. Recently, the domain adversarial
network was proposed to reduce domain mismatch of training data and learn
domain-invariant features. It is thus worth exploring whether adversarial
training can further promote knowledge sharing in multilingual models. In this
work, we apply the domain adversarial network to encourage the shared layers of
a multilingual model to learn language-invariant features. Bidirectional Long
Short-Term Memory (LSTM) recurrent neural networks (RNN) are used as building
blocks. We show that shared layers learned this way contain less language
identification information and lead to better performance. In an automatic
speech recognition task for seven languages, the resultant acoustic model
improves the word error rate (WER) of the multilingual model by 4% relative on
average, and the monolingual models by 10%.
