Loss jumps down when I restart the training from a checkpoint
Hello! I’m trainingroberta-basefor many hours with the huggingfaceTrainer. Sometimes I had to manually stop and then restart the training usingtrainer.train(resume_from_checkpoint=True)for issues related to the gpus I am using. The problem is that every time that I restart the training, I can see the loss jumping down both on validation and training set. See my wandb plot:Have anyone had the same problems or do you know how to fix it?
2024-04-26T11:51:01Z
Hi everyone, I’m new here. I was looking for a solution to this problem and came across this topic, I hope someone can help solve it. Good luck to everyone.
2024-04-28T03:00:30Z
Probably it is due to the fact the learning rate decreases along the epochs. when you restart, it is on its maximum value, causing the drop loss. It happens in many frameworks. Not sure about your settings
2024-04-29T07:46:12Z
I don’t think this is the problem. Look at the learning rate decay, it seems to go to 0 linearly.image1272×808 21.7 KB