Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers
patrickvonplaten
<a target="_blank" href="https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb">
    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

Wav2Vec2 is a pretrained model for Automatic Speech Recognition (ASR)
and was released in [September
2020](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/)
by Alexei Baevski, Michael Auli, and Alex Conneau.

Using a novel contrastive pretraining objective, Wav2Vec2 learns
powerful speech representations from more than 50.000 hours of unlabeled
speech. Similar, to [BERT\'s masked language
modeling](http://jalammar.github.io/illustrated-bert/), the model learns
contextualized speech representations by randomly masking feature
vectors before passing them to a transformer network.

![wav2vec2\_structure](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/wav2vec2.png)

For the first time, it has been shown that pretraining, followed by
fine-tuning on very little labeled speech data achieves competitive
results to state-of-the-art ASR systems. Using as little as 10 minutes
of labeled data, Wav2Vec2 yields a word error rate (WER) of less than 5%
on the clean test set of
[LibriSpeech](https://huggingface.co/datasets/librispeech_asr) - *cf.*
with Table 9 of the [paper](https://arxiv.org/pdf/2006.11477.pdf).

In this notebook, we will give an in-detail explanation of how
Wav2Vec2\'s pretrained checkpoints can be fine-tuned on any English ASR
dataset. Note that in this notebook, we will fine-tune Wav2Vec2 without
making use of a language model. It is much simpler to use Wav2Vec2
without a language model as an end-to-end ASR system and it has been
shown that a standalone Wav2Vec2 acoustic model achieves impressive
results. For demonstration purposes, we fine-tune the \"base\"-sized
[pretrained checkpoint](https://huggingface.co/facebook/wav2vec2-base)
on the rather small [Timit](https://huggingface.co/datasets/timit_asr)
dataset that contains just 5h of training data.

Wav2Vec2 is fine-tuned using Connectionist Temporal Classification
(CTC), which is an algorithm that is used to train neural networks for
sequence-to-sequence problems and mainly in Automatic Speech Recognition
and handwriting recognition.

I highly recommend reading the blog post [Sequence Modeling with CTC
(2017)](https://distill.pub/2017/ctc/) very well-written blog post by
Awni Hannun.

Before we start, let\'s install both `datasets` and `transformers` from
master. Also, we need the `soundfile` package to load audio files and
the `jiwer` to evaluate our fine-tuned model using the [word error rate
(WER)](https://huggingface.co/metrics/wer) metric \\({}^1\\).

```bash
!pip install datasets>=1.18.3
!pip install transformers==4.11.3
!pip install librosa
!pip install jiwer
```

Next we strongly suggest to upload your training checkpoints directly to the [Hugging Face Hub](https://huggingface.co/) while training. The Hub has integrated version control so you can be sure that no model checkpoint is getting lost during training. 

To do so you have to store your authentication token from the Hugging Face website (sign up [here](https://huggingface.co/join) if you haven't already!)

```python
from huggingface_hub import notebook_login

notebook_login()
```

**Print Output:**
```bash
Login successful
Your token has been saved to /root/.huggingface/token
Authenticated through git-crendential store but this isn't the helper defined on your machine.
You will have to re-authenticate when pushing to the Hugging Face Hub. Run the following command in your terminal to set it as the default

git config --global credential.helper store
```

Then you need to install Git-LFS to upload your model checkpoints:

```python
!apt install git-lfs
```