Settings warmup_steps=1 dosn't fixed the learning rate , which change every epcohs
I’m training model with the following parameters:Seq2SeqTrainingArguments(
    output_dir                   = "./out", 
    overwrite_output_dir         = True,
    do_train                     = True,
    do_eval                      = True,
    
    per_device_train_batch_size  = 2, 
    gradient_accumulation_steps  = 4,
    per_device_eval_batch_size   = 8, 
    
    learning_rate                = 1.25e-5,
    warmup_steps                 = 1,
    
    save_total_limit             = 1,
       
    evaluation_strategy          = "epoch",
    save_strategy                = "epoch",
    logging_strategy             = "epoch",  
    num_train_epochs             = 5,   
    
    gradient_checkpointing       = True,
    fp16                         = True,    
        
    predict_with_generate        = True,
    generation_max_length        = 225,
          
    report_to                    = ["tensorboard"],
    load_best_model_at_end       = True,
    metric_for_best_model        = "wer",
    greater_is_better            = False,
    push_to_hub                  = False,
)After finished to trained, I’m looking on the filetrainer_state.json, and it seems that the learning rate is not fixed.Here are the values oflearning_rateandstep:learning_rate,     steps1.0006 e-05       1033
7.5062 e-06       2066
5.0058 e-06       3099
2.5053 e-06       4132
7.2618 e-09       5165It seems that the learning rate is not fixed on1.25e-5(after step 1)What am I missing ?
2024-01-10T21:36:06Z
Hi@laro1, try passinglr_scheduler_type="constant_with_warmup"to your training args. The default is"linear", which means it’ll decay linearly after warmup
2024-01-11T11:36:42Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.