Wav2vec-C: A Self-supervised Model for Speech Representation Learning
  Wav2vec-C introduces a novel representation learning technique combining
elements from wav2vec 2.0 and VQ-VAE. Our model learns to reproduce quantized
representations from partially masked speech encoding using a contrastive loss
in a way similar to Wav2vec 2.0. However, the quantization process is
regularized by an additional consistency network that learns to reconstruct the
input features to the wav2vec 2.0 network from the quantized representations in
a way similar to a VQ-VAE model. The proposed self-supervised model is trained
on 10k hours of unlabeled data and subsequently used as the speech encoder in a
RNN-T ASR model and fine-tuned with 1k hours of labeled data. This work is one
of only a few studies of self-supervised learning on speech tasks with a large
volume of real far-field labeled data. The Wav2vec-C encoded representations
achieves, on average, twice the error reduction over baseline and a higher
codebook utilization in comparison to wav2vec 2.0
