How does `max_steps` affect the number of samples the model "sees"?
Hello,I am finetuningstarcoderbase-3bwith a small local dataset that I have with around ~7000 programs. I’ve looked online, but I am still confused as to how themax_stepsparameter affects how many times the model sees each sample (program).If I setmax_steps = 21000will the model see each program 3 times? I.e. something like 3 epochs of training? If I setmax_steps < 7000will the modelnotsee all the programs? And I am asking this, because no matter themax_stepsvalue, at the end of the training I seem to get 1epoch.Currently, I have abatch_size = 1andgradient_accumulation_steps = 4? Do those numbers also play a role into this?Thank you
2024-01-18T20:57:54Z
Hi@gsakkas,The max steps arg is referring to  the number ofoptimizationsteps that run during training. That is, the number of forward + backward passes. Because forward/backward passes are batched, batch size and gradient accumulation do factor into it.If you have 7000 examples, batch size = 1, and gradient accumulation = 4, then in order for training to see every example just once, you need 7000 / (4 * 1) = 1750 steps. If you had a batch size of 8 and gradient accumulation of 4, you’d need 219 steps for the model to see every example once.So given your config, 21000 steps would mean the model sees every example 12 times.It’s often less confusing to set the number ofepochsinstead of the number of steps. 1 epoch means that no matter what the batch size/gradient accumulation is, every example will be seen just once. 2 epochs, twice, etc.
2024-01-18T21:50:05Z
Thank you for your elaborate response!Just to clarify, when I setmax_steps, even with a higher value that, as you said, is enough for 12 epochs, the reported epoch and the end of the training seems to be always 1. Is the true number of epochs not reported whenstepsare used? I think my confusion stemmed from that.
2024-01-19T01:46:31Z
I’m not sure what’s going on in that case. My read of the trainer code is thatnum_epochsshould be set to 12 (looking atthis part). Whenthislogger statement is reached when you train, does it just say 1?
2024-01-19T13:46:41Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.