Linear Learning Rate Warmup with step-decay
Hey Guys,I am new to PyTorch and have to train deep models like ResNet-, VGG-which requires the following learning rate schedule:Linearly increase the learning rate from 0 to ‘initial_lr’ in the first k training steps/iterationsContinue with ‘initial_lr’ for the next ‘m’ training stepsDecay the learning rate in a step-decay manner. For example, say after 30th epoch, you reduce the ‘initial_lr’ by 10. And after 45th epoch, you further reduce it by 10 for any further training.This can be better visualized using the following picture:This is an example using LeNet-300-100 on MNIST with TensorFlow2.How can I achieve this particular learning rate schedule with ‘huggingface’?Thanks
2021-04-19T07:42:20Z
hey@adaptivedecay, you can define your own scheduler for the learning rate by subclassingTrainerand overriding thecreate_schedulerfunction to include your logic:Trainer — transformers 4.5.0.dev0 documentationalternatively, you can pass the optimizer and scheduler as a tuple in theoptimizersargument.
2021-04-19T07:59:01Z
Hi@lewtun, any sample code would be helpful?
2021-04-19T08:21:08Z
you can take a look at the implementation in theTrainerhere:transformers/trainer.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub
2021-04-21T13:22:43Z
Thanks for your link. I will have a look into it and get back!