Resume Training with Lower Learning Rate
I’m training a model which started to diverge during the warmup stage of training. I would like to continue training from the last stable checkpoint with a lower learning rate.I tried passing a new learning rate toTrainingArguments, but the new learning rate is overwritten when the checkpoint is loaded. Currentlytrainer.train(resume_from_checkpoint='checkpoint')will overwrite the new learning rate with the old learning rate.How can I resume training from a checkpoint, skip the batches already trained on using the same RNG seed, but also use a lower learning rate? Is this even possible without doing something hacky with theTrainercode?Additionally, I spend a significant amount of time trying to resume training with different hyperparameters before realizing the new params were being overwritten byresume_from_checkpoint. This behavior is not obvious from a user perspective and should be much more explicit.
2024-07-19T01:23:00Z
Same problem, anyone help?
2025-01-04T16:23:48Z
Same here. 1k views and no answer yet
2025-01-05T19:00:33Z
To resume training from a checkpoint with a lower learning rate while maintaining the same RNG seed and skipping already trained batches, you have a few options:Modify the checkpoint: Before loading, you could manually edit the checkpoint file to update the learning rate. This approach is somewhat hacky and requires understanding the checkpoint file structure.Override after loading: Immediately after loading the checkpoint but before resuming training, you could manually override the optimizer’s learning rate. This method is less intrusive but requires careful timing in your code.Custom checkpoint loading: Implement a custom checkpoint loading function that restores the model state and optimizer state separately, allowing you to modify the learning rate before fully reconstructing the optimizer.Use a learning rate scheduler: Instead of directly setting a lower learning rate, use a learning rate scheduler that reduces the rate after resuming from the checkpoint.Regarding the issue with hyperparameters being overwritten, you’re right that this behavior should be more explicit. Here are some suggestions to improve the user experience:Add clear documentation: Explicitly state in theresume_from_checkpointfunction documentation that it overwrites training arguments.Implement a warning system: Add a warning message whenresume_from_checkpointoverwrites user-specified hyperparameters.Provide an option: Add a parameter toresume_from_checkpointthat allows users to choose whether to use checkpoint hyperparameters or newly specified ones.Return overwritten values: Haveresume_from_checkpointreturn a dictionary of any overwritten values, allowing users to easily see what changed.Two-step process: Split the process into loading the checkpoint and applying the hyperparameters, giving users more control over what gets overwritten.These changes would make the behavior more transparent and give users more control over the resumption process