Questions about default checkpointing behavior (train v. val)
Hello,I had a few questions about how Huggingface checkpoint behavior changes depending on the arguments to the Trainer.In the documentation, I noticed that by default:because evaluation_strategy is ‘no’, evaluation is never run during training. Reference was here:Trainer — transformers 4.7.0 documentationbecause metric_for_best_model is by default ‘None’, the metric defaults to “loss”, which is the same as “eval_loss”. The same reference as above.My questions were:If I run a model with default arguments, does the model checkpointing automatically save and load the best checkpoint based on validation loss/the eval_dataloader?Do the losses displayed in trainer_state.json correspond to val or train loss?Is there an easy way to plot the train and val losses that doesn’t involve overriding the default model behavior or going through an external visualization library like comet?Additionally, I tried manually overwriting the metric_for_best_model as follows:training_args.metric_for_best_model = “eval_loss”I do this before the training arguments are passed to Trainer initialization.If I do this, 1) is this a correct way to enforce validation-based checkpointing? and 2) in this situation, what are the losses displayed in trainer_state.json?Thank you for your help! I appreciate it.
2021-08-10T08:20:12Z
Hi there, here are the answers:No, by default the model checkpointing only saves model to resume training later if something goes wrong, but there is no best model loading logic unless you useload_best_model_at_end. You will then need to set aneval_strategyand asave_strategythat match (either epoch or steps)It’s the accumulated training loss since the beginning of the training.No there is not inside Trainer. We integrate with most reporting tooling, TensorBoard, Wandb, CometML etc for this reason.For your last questions, settingmetric_for_best_modelis not enough, you need to setload_best_model_at_endtoTrue. The losses dispalyed in thetrainer_state.jsonwill still be the training losses.
2021-08-23T15:08:47Z
Thank you for your help!If I have load_best_model_at_end activated when I train my model, whenever I use theBertForMaskedLM.from_pretrained()am I guaranteed to automatically recover my best checkpoint?
2021-08-30T18:21:45Z
If you use thefrom_pretrainedmethod you will get the model associated with the folder/model identifier you pass. This class has no knowledge of the best checkpoint.
2023-10-16T13:54:09Z
hi,is it possible to save model base on the best traning loss without the need to passeval_strategy?