Wav2Vec2: Inner workings of the Trainer class
Hi all, I am following thisguidein order to fine-tune the model for my dataset. Reading the documentation ofWav2Vec2ForCTC, it says that the argumentattention_maskmust only be passed when the model’s processor hasconfig.return_attention_mask == True. The processor that is created in the blog post, has indeed itsreturn_attention_maskargument set toTrue.When setting up the trainer, does it take care of this by itself? Does it understand what arguments to use based on the settings? Or do I need to instruct it somehow to do it? By the way, as far as I understand theattention_maskis needed only when I input abatchof data into the model, not for single data points.Thanks in advance.
2021-08-30T18:17:58Z
The Trainer takes the datasetsafterpreprocessing has been applied, so setting this has nothing to do with the Trainer class.
2021-08-30T19:52:21Z
Thank you for the answer. If you notice in the blog post, theattention_maskfeature is not included in the final dataset (the one after the preprocessing is done). Therefore, is it correct to assume that we do not pass the argumentattention_maskin the model after all, even though we should?Nevertheless, in theDataCollatorclass, we use the value-100in the arrays that contain the labels of the input, to indicate what parts correspond to padding tokens. Is this something equivalent?Thanks in advance.
2021-08-30T19:54:02Z
Yes, the -100 indicates to the loss function the corresponding tokens should be ignored in the loss computation.
2021-08-30T20:00:49Z
Good to know. I suspected they were equivalent options but I was not certain. I noticed that during thetest setevalutation, theattention_maskis indeed passed due to the fact that there are no-100values in this case (since there is not dataloader/data collator in the test set case).Thanks for the answers.
2021-08-30T20:11:57Z
I thought about this a bit now, and I have one more question if you can answer: the attention mask takes care of what tokens the model attends to. If we do not pass theattention_maskto the model, does the-100value takes cares of that as well?As you said, it indicates to the loss function what tokens to ignore. But the decision to attend to specific tokens or not is something different, isn’t it?
2021-09-06T17:26:34Z
After researching this in the docs, I found out the following things:In the blog post I have provided, there is a Data Collator class that is used. This class is responsible for preparing a batch before using it as input for the model. By replicating the inner workings of this class, I realized that the batch it returnsdoesinclude the featureattention_mask, even though it is not included in the original dataset that is fed to this Data Collator class.After this, by looking in the docs for theTrainer classand in the docs that show the variousTrainer class utilities, in the methodtrain()of the first one can see that the model is iterating over all the batches provided by a PyTorch dataloader. Each batch is passed into the model (see the methodcompute_loss()in the first link above), by the following line of code:model(**inputs).Note that theinputsvariable is one batch provided to us by the dataloader, which is a dictionary that includes theattention_maskkey, as mentioned in my first point. This key is there, and it is not removed anywhere, which leads me to assume that the linemodel(**inputs)implicitly passes theattention_maskin the model, as requested by the official docs of theWav2Vec2ForCTC.This is what I’ve found, and I’d really appreciate it if someone much more experienced than me can confirm these results.