Abnormal learn rate curve
I am working on a school project which is to classify news headlines. It’s a binary classification. I scraped the news headlines, used sklearn train_test_split to split them. Then usedktrain- distilBert to classify them. There is a learn rate finder function, I run that and get an abnormal learn rate curve as shown in below image:Screenshot 2021-01-01 at 13.54.54792×564 28.3 KBwhile the normal learn rate should be somehow in a U-shape, falls gradually from a higher loss then up again.What does that abnormal learn rate curve imply? Is it to do with overfitting or anything? I am really new to the transformer thing and there are not many resources on the internet so I try to ask here. Thanks.
2021-01-02T09:04:46Z
Hi, not sure if I understand correctlyWhich “learn rate finder function” you use ? ( just curious as I am familiar of the idea coming from fast.ai team )spike in 10^-1 looks plausible to me since it’s too much big learning rateDid you initialize your model from some checkpoints ? If yes, maybe the model is already good and have small loss in the beginning, so small loss with LR=10^-7 is also plausible
2021-01-02T11:02:34Z
I just follow this step by step tutorial ofktrain, it is a light weight wrapper of huggingface transformer. It has the learn rate finder function. I am not sure about the initializing from checkpoint part, does that mean a pretrained model? As far as I know, the tutorial is using the pretrained distilBert from Huggingface.
2021-01-02T11:06:18Z
Also, the learn rate curve demonstrated in the tutorial is like this:Screenshot 2021-01-01 at 13.56.16788×566 40.2 KBThat’s why I think mine is not a normal one.
2021-01-02T11:56:28Z
Yes, Pretrained model (= checkpoint i mentioned above) can be one reason. Maybe you can try remove the pretrained (init from scratch) and plot the graph again
2021-01-02T16:17:37Z
Note that the learning rate curve that you post seems to benotthe actual lr that is used during training, but a utility function that tests out different lr’s to see how that influences the loss function. Its goal is to help you find a good starting lr. In other words, the graph is not the learning rate changing over time, but the loss over different a different lr parameter. In that sense, it is very normal that that curve is different between checkpoints, architectures, and even datasets.
2021-01-02T21:20:32Z
Cool to know these, thanks both!