Do you train all layers when fine-tuning T5?
From my beginner-level understanding, when it comes to BERT, sometimes people will train just the last layer, or sometimes they’ll train all layers a couple epochs and then the last layer a few more epochs.Does T5 have any similar practices? Or is it normal to just train the whole thing when fine-tuning?And very tangentially related: to fine-tune T5, we just doloss.backward()on the result of theforward()call of the T5 model right (thelosskey in the returned dict)? So there’s no need to calculate any loss on our own?
2020-09-08T06:04:39Z
I haven’t seen much experiments for this, but IMO it’s better to fine-tune the whole model.Also when you passlabelsargument toT5ForConditionalGeneration'sforwardmethod then it calculates the loss for you and returns it as the first value in the returnedtuple.And you can use thefinetune.pyscripthereto fine-tuning T5 and other seq2seq modelsSee this threadT5 Finetuning Tips
2021-09-29T12:23:34Z
Hi,I am trying to fine tune T5 for conditional text generation, and it works perfectly. But, I would like to have more control over the layers I am training i.e. freeze and unfreeze certain layers. For example, I am also using GPT2 for the same use case and I do something like this:# - Freeze selective layers:
# - Freeze all layers except last n:
for parameter in model.parameters():
    parameter.requires_grad = False

for i, m in enumerate(model.transformer.h):        
    #Only un-freeze the last n transformer blocks
    if i+1 > 12 - config['UNFREEZE_LAST_N']:
        for parameter in m.parameters():
            parameter.requires_grad = True 

for parameter in model.transformer.ln_f.parameters():        
    parameter.requires_grad = True

for parameter in model.lm_head.parameters():        
    parameter.requires_grad = TrueNow, I am able to freeze all the layers in T5, but I have trouble unfreezing the last n layers, since T5ForConditionalGeneration has no attribute “transformer”. Does anyone has an idea on how to achieve this, because I couldn’t find anything about this issue?
2021-09-30T15:34:17Z
Maybe start this as a fresh topic with a more specific title to your issue - likely to stand more chance of a response? (sorry can’t help, just my experience of this forum!).
2021-10-01T07:58:00Z
You can print out any PyTorch model to see its architecture, or you can simply look at the source code. On the highest level you’ll find shared embeddingsshared, an encoderencoder, a deocderdecoderand a LM headlm_head. You can use any of those. So for instance, unfreezing the decoder transformer layers:for i, m in enumerate(model.decoder.block):        
    #Only un-freeze the last n transformer blocks in the decoder
    if i+1 > 12 - config['UNFREEZE_LAST_N']:
        for parameter in m.parameters():
            parameter.requires_grad = TrueIt isblockbecause thedecoderis aT5Stackwhich has its transformer layers as a ModuleList calledblock.github.comhuggingface/transformers/blob/8bbb53e20b7873ba7f63be70d4d798e0c3568bfa/src/transformers/models/t5/modeling_t5.py#L811return shifted_input_idsclass T5Stack(T5PreTrainedModel):def __init__(self, config, embed_tokens=None):super().__init__(config)self.embed_tokens = embed_tokensself.is_decoder = config.is_decoderself.block = nn.ModuleList([T5Block(config, has_relative_attention_bias=bool(i == 0)) for i in range(config.num_layers)])self.final_layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)self.dropout = nn.Dropout(config.dropout_rate)self.init_weights()# Model parallelself.model_parallel = Falseself.device_map = Noneself.gradient_checkpointing = FalseSo it isn’t hard at all, you just have to go look for the right name in the code.
2021-10-01T11:56:15Z
Thank you, this was actually very helpful
2023-09-26T07:48:18Z
Do you guys know what the default behavior is? Does finetuning unfreeze all layers and weights? Or does it just unfreeze the LM head (linear projection)?
2023-09-26T10:45:43Z
The default behavior in Huggingface is to fine-tune the whole model, there’s no freezing. If you want to freeze layers, you actually have to write some code yourself to do that.