Collapsing Wav2Vec2 pretraining loss
I’m trying to pretrain a Wav2Vec2 model based on the example givenhere`.I was initially getting a contrastive loss like the graph on the left which seemed very slow so I upped the learning rate and got the graph on the right after only a few steps.image880×426 33.7 KBI’m not familiar with the nuts and bolts of contrastive loss but this came as a bit of a surprise and I was wondering if anyone could help me understand.The batch size (with accumulation) is 32, the number of epochs is 20 and the warmup steps is 1200 for both attempts.
2021-09-20T10:57:45Z
The solution in the end was to setreturn_attention_masktoTruein the feature extractor, or use a pretrained feature extractor and model that prefers attention masks (i.e. notwav2vec2-base).
2023-04-03T11:56:20Z
Hi, thank you for sharing your experience and solution. Concerning the attention mask, do you input to the model directly the attention mask given by the feature extractor (at sample level and not at frame level right?) ? Are you able to check the shapes?Thanks