How to freeze layers using trainer?
Hey,I am trying to figure out how to freeze layers of a model and read that I had to usefor param in model.base_model.parameters():
    param.requires_grad = Falseif I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer?I tried the following:from transformers import BertTokenizer, BertForMaskedLM. LineByLineTextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

for param in model.base_model.parameters():
    param.requires_grad = False

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path=in_path,
    block_size=512,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir=out_path,
    overwrite_output_dir=True,
    num_train_epochs=25,
    per_device_train_batch_size=48,
    save_steps=500,
    save_total_limit=2,
    seed=1
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset
)

trainer.train()If the encoder was frozen I would expect it to produce the same outputs as a fresh instance of the pretrained encoder, but it doesn’t:model_fresh = BertForMaskedLM.from_pretrained('bert-base-uncased')
inputs = tokenizer("This is a boring test sentence", return_tensors="pt")
torch.all(model.bert(**inputs)[0].eq(model_fresh.bert(**inputs)[0]))
--> tensor(false)So I must be doing somethin wrong here, I guess the Trainer is reseting the requires_grad attribute and I have to overwrite it somehow after I instanciated the trainer?Thanks in advance!Johannes
2021-03-19T12:56:04Z
Looking at thesource code of BertForMaskedLM, the base model is the “bert” attribute, not the “base_model” attribute. So if you want to freeze the parameters of the base model before training, you should typefor param in model.bert.parameters():
    param.requires_grad = Falseinstead.
2021-03-19T12:58:18Z
@nielsrbase_modelis an attribute that will work on all thePreTraineModel(to make it easy to access the encoder in a generic fashion)TheTrainerputs your model into training mode, so your difference might simply come from that (there are dropouts in the model). You should check if putting it back in eval mode solves your problem.
2021-03-19T12:59:04Z
@sguggeroh didn’t know that, I learn every day!
2021-03-19T13:01:38Z
OMG! This is so obvious and I cant believe I didn’t realize that. Will test and report! Thanks
2021-03-19T16:13:03Z
@sguggermodel.eval() should have done the trick, right? I am afraid the results still don’t match
2021-03-19T16:37:39Z
You should inspect the weights to see where they difer then.Trainerwill not change therequires_gradvalue of your parameters.
2021-03-22T10:53:59Z
@sguggerThanks, that was important to know for me so I knew I had to be the one screwing up somewhere else and I did somehow manage that
2021-07-14T19:55:10Z
Hi,I tried your code, but I am getting this error:AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'
2021-07-14T20:26:00Z
Hey,yeah, this is because you are using roberta instead of bert, therefore it uses .roberta to store the encoder. I believe there is some model independant keyword like “base_model” or something, but I dont know right now (im on vacation, but maybe you can try or google it). Hope that helps!BestJohannes
2021-07-14T20:30:30Z
Sorry, answered per mail, sgugger literary provided the base_model keyword in this thread, so there you go
2024-05-26T17:55:16Z
Maybe its a problem with using eq instead of isclose?