Why my training loss drops at epoch boundaries?
I am training GPT-J with DeepSpeed. The training loss kept decreasing, which is good, but the validation loss starts climbing as early as 2 epochs. However, the oddest thing is that the training loss drop and validation loss jump seem synchronized, and always happen at the boundary of each epoch. It doesn’t look right, but I have no clue what’s the cause. Does anyone have a suggestion what to investigate? I tried to add 10 steps of warmup or reducing learning rate from the default 5e-5 to 2e-2. But no changes. I always see such “stair” shape learning curve.image867×531 75.9 KB
2022-09-16T11:15:44Z
Hi@dunalduck0are you using TensorFlow or PyTorch out of curiosity?I know this post is old, but did you manage to find an answer in the end? I’m facing a similar issue
2024-07-18T09:19:38Z
Increasing the dataset or reducing the learning rate or turning off the train shuffle can alleviate or eliminate this phenomenon.
2024-08-21T15:00:59Z
Thanks for you advice.Turn off the shuffle for train dataloader did work for me.But I wonder why it works.Maybe turning off shuffle will provides better data distribution? I used to conclude this loss drop as memorization till I turn off shuffle as you advised.It is really confused to me.
2024-08-22T14:39:20Z
Thanks - turning off train shuffle solved it for me also.