Early stopping callback problem
Hello,I am having problems with the EarlyStoppingCallback I set up in my trainer class as below:training_args = TrainingArguments(
    output_dir = 'BERT',
    num_train_epochs = epochs,
    do_train = True,
    do_eval = True,
    evaluation_strategy = 'epoch',
    logging_strategy = 'epoch',
    per_device_train_batch_size = batch_size,
    per_device_eval_batch_size = batch_size,
    warmup_steps = 250,
    weight_decay = 0.01,
    fp16 = True,
    metric_for_best_model = 'eval_loss',
    load_best_model_at_end = True
)

trainer = MyTrainer(
    model = bert,
    args = training_args,
    train_dataset = train_dataset,
    eval_dataset = val_dataset,
    compute_metrics = compute_metrics,
    callbacks = [EarlyStoppingCallback(early_stopping_patience = 3)]
)

trainer.train()I keep getting the following error:Screen Shot 2021-04-21 at 09.45.06954×561 78.7 KBI already tried running the code without themetric_for_best_modelarg, but it still gives me the same error.I tweaked the Trainer class a bit to report metrics during training, and also created custom_metrics to report during validation. I suspect that maybe I made a mistake there and that’s why I can’t retrieve the validation loss now.See herefor the tweaked code.Thanks in advance!!
2021-04-21T12:09:30Z
You won’t be able to use theEarlyStoppingCallbackwith a nested dictionary of metrics as you did, no. And is will need the metric you are looking for to be prefixed byeval_(otherwise it will add it unless you change the code too). You probably will need to write your own version of the callback for this use case.At some point, instead of rewriting the wholeTrainer, you might be interested in writing your own training loop withAccelerate. You can still have mixed precision training and distributed training but will have full control over your training loop. There is one example for each task using accelerate (therun_xxx_no_trainer) in theexamplesof Transformers
2021-04-22T13:43:37Z
Thanks so much@sgugger! Will try it out!