Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR
  in Transfer Learning
  In this work, we study leveraging extra text data to improve low-resource
end-to-end ASR under cross-lingual transfer learning setting. To this end, we
extend our prior work [1], and propose a hybrid Transformer-LSTM based
architecture. This architecture not only takes advantage of the highly
effective encoding capacity of the Transformer network but also benefits from
extra text data due to the LSTM-based independent language model network. We
conduct experiments on our in-house Malay corpus which contains limited labeled
data and a large amount of extra text. Results show that the proposed
architecture outperforms the previous LSTM-based architecture [1] by 24.2%
relative word error rate (WER) when both are trained using limited labeled
data. Starting from this, we obtain further 25.4% relative WER reduction by
transfer learning from another resource-rich language. Moreover, we obtain
additional 13.6% relative WER reduction by boosting the LSTM decoder of the
transferred model with the extra text data. Overall, our best model outperforms
the vanilla Transformer ASR by 11.9% relative WER. Last but not least, the
proposed hybrid architecture offers much faster inference compared to both LSTM
and Transformer architectures.
