Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers
patrickvonplaten
<a target="_blank" href="https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLS_R_on_Common_Voice.ipynb">
    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>


***New (11/2021)***: *This blog post has been updated to feature XLSR\'s
successor, called [XLS-R](https://huggingface.co/models?other=xls_r)*.

**Wav2Vec2** is a pretrained model for Automatic Speech Recognition
(ASR) and was released in [September
2020](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/)
by *Alexei Baevski, Michael Auli, and Alex Conneau*. Soon after the
superior performance of Wav2Vec2 was demonstrated on one of the most
popular English datasets for ASR, called
[LibriSpeech](https://huggingface.co/datasets/librispeech_asr),
*Facebook AI* presented a multi-lingual version of Wav2Vec2, called
[XLSR](https://arxiv.org/abs/2006.13979). XLSR stands for *cross-lingual
speech representations* and refers to model\'s ability to learn speech
representations that are useful across multiple languages.

XLSR\'s successor, simply called **XLS-R** (refering to the
[*\'\'XLM-R*](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/)
*for Speech\'\'*), was released in [November 2021](https://ai.facebook.com/blog/xls-r-self-supervised-speech-processing-for-128-languages) by *Arun
Babu, Changhan Wang, Andros Tjandra, et al.* XLS-R used almost **half a
million** hours of audio data in 128 languages for self-supervised
pre-training and comes in sizes ranging from 300 milion up to **two
billion** parameters. You can find the pretrained checkpoints on the 🤗
Hub:

-   [**Wav2Vec2-XLS-R-300M**](https://huggingface.co/facebook/wav2vec2-xls-r-300m)
-   [**Wav2Vec2-XLS-R-1B**](https://huggingface.co/facebook/wav2vec2-xls-r-1b)
-   [**Wav2Vec2-XLS-R-2B**](https://huggingface.co/facebook/wav2vec2-xls-r-2b)

Similar to [BERT\'s masked language modeling
objective](http://jalammar.github.io/illustrated-bert/), XLS-R learns
contextualized speech representations by randomly masking feature
vectors before passing them to a transformer network during
self-supervised pre-training (*i.e.* diagram on the left below).

For fine-tuning, a single linear layer is added on top of the
pre-trained network to train the model on labeled data of audio
downstream tasks such as speech recognition, speech translation and
audio classification (*i.e.* diagram on the right below).

![wav2vec2\_structure](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/xls_r.png)

XLS-R shows impressive improvements over previous state-of-the-art
results on both speech recognition, speech translation and
speaker/language identification, *cf.* with Table 3-6, Table 7-10, and
Table 11-12 respectively of the official [paper](https://ai.facebook.com/blog/xls-r-self-supervised-speech-processing-for-128-languages).

Setup