Domain Adaptation via Teacher-Student Learning for End-to-End Speech
  Recognition
  Teacher-student (T/S) has shown to be effective for domain adaptation of deep
neural network acoustic models in hybrid speech recognition systems. In this
work, we extend the T/S learning to large-scale unsupervised domain adaptation
of an attention-based end-to-end (E2E) model through two levels of knowledge
transfer: teacher's token posteriors as soft labels and one-best predictions as
decoder guidance. To further improve T/S learning with the help of ground-truth
labels, we propose adaptive T/S (AT/S) learning. Instead of conditionally
choosing from either the teacher's soft token posteriors or the one-hot
ground-truth label, in AT/S, the student always learns from both the teacher
and the ground truth with a pair of adaptive weights assigned to the soft and
one-hot labels quantifying the confidence on each of the knowledge sources. The
confidence scores are dynamically estimated at each decoder step as a function
of the soft and one-hot labels. With 3400 hours parallel close-talk and
far-field Microsoft Cortana data for domain adaptation, T/S and AT/S achieve
6.3% and 10.3% relative word error rate improvement over a strong E2E model
trained with the same amount of far-field data.
