Training on Domain specific Dataset
Hi ,I want to train a sentiment analysis multi-label classifier and in addition to training the final output layers, I’d like to train the hidden BERT layers as well.I want to understand how much improvement can I get in my metric(F1 score) by feeding it my domain-specific data.All the documents /references I have seen thus far only point to training the final output layer that generates classification. Is there a way to train various hidden layers of BERT using (let’s say) BERT Base?Thanks in advance,Devesh
2021-03-21T09:24:37Z
My understanding is that if you don’t specifically freeze any of the layers you will always train the whole model.If you want to train only particular layers, you can add a condition to this code:model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False
2021-03-21T17:34:14Z
Hi@neuralpat,Thanks so much for taking the time and responding to my queries.I’m going to try it out.I am also wondering if there is a way to freeze specific layers(eg. Top 3 ) and train ?
2021-03-22T21:15:20Z
Hi,model = BertForSequenceClassification.from_pretrained('bert-base-uncased')for param in model.bert.parameters():
    param.requires_grad = Falseas far I understand, the code above is forALLparamsbut you could easily limit to 3 elements :model.bert.parameters()is a generatori=0
for i,el in enumerate(model.parameters()):
  if i<3:
    el.requires_grad=False
  else:
    el.requires_grad=TrueYou can check the result with the result with:for param in model.parameters():
    print(param)