Fine-Tune Wav2Vec2 for English ASR with 🤗 Transformers article bug
Hello@patrickvonplaten!Thank you so much for the tutorials. Super helpful!I’ve been having troubles with setting up a CTC head on a pre-trained model with an existing CTC head and wanted to point out a possible problemin the tutorialthat in part led me to having my main problem.First, I’ll pinpoint the problem in the tutorial, and then I’ll describe the problem I am getting. I am not that concerned with my problem yet as I haven’t dived deep into the source code to actually find what is wrong. I just want to make sure I am not crazy with the tutorial thing.TUTORIALSo, first, we’re setting up a vocabulary of 30 tokens and loading it into a tokenizer. It is indeed 30 tokens.Then, we proceed to load a pre-trained model with CTC head initialized at random. The thing is, the head is initialized with the size of the vocabulary being 32! And we never correct it. Are those supposed to be bos and eos? Shouldn’t we add those to the vocab?Screenshot 2021-10-01 at 17.02.181398×976 91.3 KBMY PROBLEMMy goal is to fine-tune a model that already has a fine-tuned CTC head to work for my data (there is no headless model). The head has more output tokens than I need. So I want to initialize it at random.I prepare my dictionary and a tokenizer, it works just as I expect it to.I load the model with the CTC head, swap it with my head. I add the vocab_size=len(tokenizer) and pad_token_id to the model config. I even add the bos_token_id and eos_token_id (and add those to the vocabulary, although I don’t need them)And then several strange things tend to happen:I was going nuts about why I have the same random character appearing between the characters that should indeed be recognized (I was printing out evaluation set predictions during training).And then I understood that this character has id 0 in the vocab I am creating. And it is usually the id for the PAD token in wav2vec vocabs.If I set the vocab size in the model config exactly to my vocab size I getraise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")ValueError: Label values must be <= vocab_size: 32so I am forced to do model.config.vocab_size=len(tokenizer)+1 for the thing to work.So if you know right away what is going wrong, I’ll appreciate the answerIf not, I am just going to dive deeper myselfKatja
2021-10-01T19:22:33Z
Well, the label error was my error with adding a special char to the vocabulary…But the addition of a token with zero id is still under my investigation
2021-10-01T22:55:35Z
I think I found it. The problem comes from torch.nn.ctc_loss. It sets blank to zero.
2022-01-11T13:27:31Z
Do you mean that the ctc always predict ‘blank’? I meet the same issue.What do you mean that" The problem comes from torch.nn.ctc_loss. It sets blank to zero"?
2022-08-26T14:58:32Z
Hi there, facing the same issue, did you manage to solve this?
2022-08-29T10:55:19Z
@patrickvonplatenwould appreciate your help here…
2022-08-30T08:09:25Z
i just set pad token to 0 in the vocab
2022-09-01T07:10:53Z
did you also have to set the len of model vocab to len of tokenizer? earlier version I trained was predicting pad token for everything
2022-10-03T02:51:16Z
I meet this issue too. But when i train model by “from transformers import AutoModelForCTC”, it is ok (transformers==4.22.2). This is link:colab.research.google.comGoogle Colaboratory
2022-11-26T21:16:51Z
This is because CTC model can only support a-z, space, [PAD], [UNK], and several other special characters, which in total are 32 characters. If your vocab_dict size is larger than 32, CTC will not be able to represent them. Hence, you have to remove some characters from the vocabulary to make its size less than 32.
2023-06-10T16:05:38Z
Hey, I’m facing the same problem currently. Model is predicting [PAD] token only. How were you able to solve this problem? Thanks!
2023-08-14T14:54:32Z
found any solution to this problem?
2023-08-17T11:04:45Z
heyyy, the problem was that the ctc loss expected a pad token to have id 0 in the token vocabulary. whenever the vocabulary had something else in that position it was used by the loss as pad token. not sure if it’s still the case. going to actually revisit it today
2023-08-22T10:46:11Z
i don’t think this problem exists today. everything works fine with any id of pad in the vocabulary.
2023-08-23T12:32:02Z
i tried this and it’s worked, thanks
2024-03-07T21:48:03Z
For people running into this problem because theirvocab_sizeis actually large than themodel.config.vocab_size(rather than the pad id not being zero), you can fix like this:model.config.update({
    "vocab_size": len(tokenizer),
})