Continue fine-tuning with Trainer() after completing the initial training process
Hey all,Let’s say I’ve fine-tuned a model after loading it usingfrom_pretrained()for40epochs. After looking at my resulting plots, I can see that there’s still some room for improvement, and perhaps I could train it for a few more epochs.I realize that in order to continue training, I have to use the codetrainer.train(path_to_checkpoint). However, I don’t know how to specify the new number of epochs that I want it to continue training for. Because it’s already finished the40epochs I initially instructed it to train for.Do I have to define a new trainer? But if I define a new trainer, can I also change the learning rate? In addition to these questions, there is also the learning rate scheduler. The default of the trainer is the OneCycleLR, if I’m not mistaken. This means that by the end of my40previous epochs, the learning rate was0. By restarting the training process, will the whole scheduler restart as well?Thanks for any help in advance.
2021-09-09T14:43:56Z
Yes, you will need to restart a new training with new training arguments, since you are not resuming from a checkpoint.The Trainer uses a linear decay by default, not the 1cycle policy, so you learning rate did end up at 0 at the end of the first training, and will restart at the value you set in your new training arguments.
2021-09-09T15:40:49Z
Hi, thanks for answering.Ah, so it’s more like restarting the training but from a check-point, not actually continuing entirely from where you left it, at least considering the learning rate’s values. I suppose I could continue the training by setting a very low learning rate, to approximate the values it would have were it to continue training normally from epoch40. Regarding the scheduler, you are right, it is the linear decay but I think it also has optional warm-up steps, in which case it becomes OneCycleLR.Also, considering the argumentoutput_dirof thenewTraininbArgumentsobjects I will define to restart training: do I pass any path I want there? I can use a different path from the previous one, or the old one. If I use the old one, is it usual to useoverwrite_output_dir=Trueso that it overwrites the old checkpoint?Thanks in advance.
2021-09-09T16:28:11Z
It depends what you want, but you can re-use the sameoutput_dirif you don’t mind overwriting your old checkpoints.
2021-09-09T16:29:48Z
Alright, thank you. Have a nice day
2021-09-09T16:58:13Z
By the way,@sgugger, in my case where I don’t want to actually continue training because the40epochs are completed, do I still pass a checkpoint path in the trainer or do I just train by just writingtrainer.train()?
2021-09-09T17:04:18Z
You should instantiate your model from the version trained, then launchtrainer.train().
2021-09-09T17:10:50Z
Great, I’ll just load it usingfrom_pretrained()and train it using new TrainingArguments. Thank you
2022-01-17T04:00:26Z
@ThomasGHello, I was facing exact the same issue and found this topic.In fact even the same number of epochs. After the 40 epochs (and 1 month, it was a big model), the learning rate reached 0 (after the warm-up and linear decay) but the loss seems that can continue to fall.Let me ask you: Do you restart training using the same learning rate as the first training? You tested other values of learning rate? How they behave? How the changes in learning rate from starting a new training impacted the loss curve? (it continued the same decay of previous training?)And at the ending of the second training, you loss loss curve keep showing that it can improve further? If so, you trained again?Thanks in advance.
2022-01-19T12:50:26Z
Hello.I just tried to simulate the behavior of the scheduler, were it to have continued training after 40 epochs. I think I set the initial LR to a value similar to the final ones of the scheduler, and I did not include a warm-up since I only wanted it to decrease. This is in no way a robust solution, it was just a though to approximate what it would do.The other questions you ask are very task-specific. Even if my loss curve showed signs that it can get better after the 2nd training ended, this does not mean yours would behave similarly. Good luck with your project!