Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired
  Speech Data
  This paper studies a novel pre-training technique with unpaired speech data,
Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within
a multi-task learning framework, we introduce two pre-training tasks for the
encoder-decoder network using acoustic units, i.e., pseudo codes, derived from
an offline clustering model. One is to predict the pseudo codes via masked
language modeling in encoder output, like HuBERT model, while the other lets
the decoder learn to reconstruct pseudo codes autoregressively instead of
generating textual scripts. In this way, the decoder learns to reconstruct
original speech information with codes before learning to generate correct
text. Comprehensive experiments on the LibriSpeech corpus show that the
proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over
the method without decoder pre-training, and also outperforms significantly the
state-of-the-art wav2vec 2.0 and HuBERT on fine-tuning subsets of 10h and 100h.
