Question Regarding trainer arguments:: load_best_model_at_end
My question is regardingtransformers.TrainingArgumentsclass argument. There are two parameter,save_total_limitload_best_model_at_endQ1. Let’s just say I have setsave_total_limit=50. But the best model found by the metric doesn’t stay in the last 50 checkpoints. Maybe it is in the last 200 checkpoints.Now shouldload_best_model_at_endwill select the best model from the last 50 checkpoints or the entire training duration?Q2. The problem regarding this is, not always we have large SSD space (or even Regular storage) to train the model. Sosave_total_limitis kind of a limited feature based on an individual’s disk space. On the contrary,save_total_limiton the best checkpoints would be a great feature. In that way, you can even look for the ensemble of multiple checkpoints (may be good for generation tasks).So is there any way you can save “best 5 checkpoints” (or best X) from the entire training duration?Note: I tried to read the source code, but too many callback functions to deal with. It would be a great time-saving if someone can help.
2021-04-19T12:58:52Z
When you useload_best_moel_at_endin combination withsave_total_limit, the checkpoint with the best metric is never deleted (it’s always put first in the list of all checkpoints).There is no way for now to keep the 5 best checkpoints, only the best one.
2021-04-19T14:23:22Z
Hi!@sguggerThanks a lot for your response. It is totally clear now.