Optimal ratio for eval_steps
I want to fine-tune the GPT-2 model on my specific dataset, with a total of 500,000 training steps and a learning rate of 5e-4. I set theeval_stepsto 5000, which means the model will be updated every 1% of the total training steps. Although, I know that the optimal frequency foreval_stepsdepends on the specific dataset and other hyperparameters, given my limited computational power, I am looking for a starting point to fine-tune the model effectively. I’ll be thankful if anyone could help me with this.