BERT: AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'
I am trying to freeze some layers of my masked language model using the following code:for param in model.bert.parameters():
    param.requires_grad = FalseHowever, when I execute the code above, I get this error:AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'In my code, I have the following imports for my masked language model, but I am unsure what is causing the error above:from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)So far, I have tried to replacebertwithmodelin my code, but that did not work.Any help would be good.Thanks.
2021-07-15T12:01:54Z
The name of the body of the modelrobertafor Roberta models, notbert. So you should loop onfor param in model.roberta.parameters(). In general, the attribute that is model agnostic isbase_model, sofor param in model.base_model.parameters()should work anywhere.
2021-07-15T14:20:58Z
Okay, thanks for that. Now, is it possible to freeze just the top layer or bottom layer of the BERT model?
2021-07-17T18:45:04Z
Yes, you can use thenames_parameters()method for that.It gives you the names of the parmeters along with the paramters themselves,so you can filter only the paramters of the top / bottom layer based on their names and freeze them.
2021-07-19T13:52:02Z
Hello, thanks for the reply. So, do I just simply add this code?names_parameters()
2021-07-19T15:29:06Z
No, you should iterate over them.Replcae the following line:for param in model.roberta.parameters():with:for name, param in model.roberta.named_parameters():.Then filter the parameters you want to freeze using thenamevariable.You may print the names to see how they look like, and then come up with a condition that will filter the ones you need.Hope it helps.
2021-07-19T16:54:56Z
Hello, thanks very much for that explanation and solution - it has really helped me quite a lot. In terms of filtering parameter, say my dataset has emojis, can I filter emojis? Like, for example, thisemoji?
2021-07-20T10:11:43Z
Amm could you please elaborate more? I’m not sure I understood your question.BTW If it is an unrelated question to the original question of this post, you may ask it in a new one.
2021-07-21T15:58:35Z
Apologies, basically I am trying to do masked language modelling using emojis, but when I deploy my model, the predicted tokens only show words, not emojis; hence, I think that the emojis are not frequent enough in the vocabulary, causing them to be less likely for the masked prediction. Therefore, I was wondering if I could freeze some layers of my BERT model to get the less frequent tokens, which are emojis, to be the top predictions when I deploy my masked language model.
2021-07-25T11:57:05Z
@Yuti- How would  I go about filtering characters ?
2021-07-25T12:24:10Z
Have you tried increasing the masking probabilty for emojis?If you have controll over the masking when training your model, you can increase the probability an emoji will be masked and then the model will output higher probabilities for emojis.I’m not sure this is a good solution but you may try it.
2021-07-25T17:03:36Z
@YutiIn my code, I have this line which allows me to adjust the masked language probability:from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)However, if I adjusted it, I am unsure how to do it for the emojis.
2021-07-25T18:12:33Z
Changing themlm_probabilityargument wont give you the result you need,but I think you can create a sub class ofDataCollatorForLanguageModelingthat does the emoji masking.You can find the source code forDataCollatorForLanguageModelinghere.