Evaluation became slower and slower during Trainer.train()
When I used Trainer.train() to fine-tune BartBase, I found something weird that the speed shown in progress bar became slower and slower (from6 item/sto0.29 item/s. Please help me, I’m new to transformers.Here are my codes.training_args = TrainingArguments(
    output_dir="Model/BartBase",
    overwrite_output_dir=True,
    
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    learning_rate=1e-5,
    num_train_epochs=20,
    lr_scheduler_type='linear',
    label_smoothing_factor=0,
    
#     logging_dir='runs',
    logging_strategy='steps', # log according to log_steps
    logging_steps=1,
    
    save_strategy='steps', # log according to save_steps
    save_steps=4000,
    save_total_limit=10, # limit the total amount of checkpoints
    
    evaluation_strategy="steps", # log according to eval_steps
    eval_steps=1, # I set eval_steps=1 to debug
    eval_accumulation_steps=1,
    
    seed=42, 
    
    load_best_model_at_end=True, # load best model according to metric_for_best_model
    metric_for_best_model='f1' # the string should be 
    )


from datasets import load_metric
import numpy as np

def compute_metrics(eval_pred):
    f1_metric = load_metric('f1')
    accuracy_metric = load_metric('accuracy')
    pred, label = eval_pred
    pred = np.argmax(pred, axis=-1)
    f1_score = f1_metric.compute(predictions=pred, references=label, average='micro')
    accuracy = accuracy_metric.compute(predictions=pred, references=label)
    return f1_socre.update(accuracy)


from transformers import Trainer
trainer = Trainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    data_collator=collator, # if tokenizer is provided, no need to provide it explicitly
    
    train_dataset=train_dataset, # torch.utils.data.dataset.Dataset
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics
)

trainer.train()
2021-07-27T08:46:45Z
After debugging step by step, I found thatIf I remove thecompute_metrics=compute_metricsin Trainer, the evaluation went well.Even if I use a quite simple compute_metrics, the evaluation became slow and stopped eventually (without finishing progress) .def compute_metrics(eval_pred): 
      return {'f1': 1}Please give me some helps. Thanks a lot!!!
2022-08-12T19:21:46Z
Hi,I am also experiencing speed issues.Did you solve it now？
2022-09-12T21:16:35Z
I was initially experiencing slower and slower performance, but upon settingpredict_with_generateto True, each batch trained with the same speed!
2022-11-17T01:22:31Z
I experience the same issue.
2023-11-20T05:35:34Z
I am experinencing the same issue!!! Anyway to fix it??
2023-11-20T07:09:57Z
! I think I found a way to solve it!According tohttps://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941/2, I think the probably reason is that “When computing metrics inside theTrainer, your predictions are all gathered together on the device (GPU/TPU) and only passed back to the CPU at the end (because that operation can be slow).”But when computing we do not need all the logits (just the largest one’s idx). So I solve the problem by introducing withpreprocess_logits_for_metricsfunction:def compute_metrics_acc(tokenizer):
        def compute_metric(eval_preds):
            preds, targets = eval_preds
            preds= np.where(preds != -100, preds, tokenizer.pad_token_id)
            targets= np.where(targets != -100, targets, tokenizer.pad_token_id)
            preds = tokenizer.batch_decode(preds, skip_special_tokens=True, clean_up_tokenization_spaces=True)
            targets = tokenizer.batch_decode(targets, skip_special_tokens=True, clean_up_tokenization_spaces=True)
            correct = 0
            assert len(preds) == len(targets)
            for idx, pred in enumerate(preds):
                reference = targets[idx]
                reference = extract_ans(reference)
                extract_pred = extract_ans(pred)
                best_option = extract_pred
                if reference == best_option and reference != False:
                    correct +=1 
            return {'accuracy': 1.0*correct/len(targets)}
        return compute_metric

def preprocess_logits_for_metrics(logits, labels):
        """
        Original Trainer may have a memory leak. 
        This is a workaround to avoid storing too many tensors that are not needed.
        """
        pred_ids = torch.argmax(logits, dim=-1)
        return pred_idsand pass it to the trainer.I left my trainer setup here:trainer = SFTTrainer(
        model=base_model,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        peft_config=peft_config,
        packing=script_args.packing,
        max_seq_length=1024,
        tokenizer=tokenizer,
        args=training_args,
        data_collator=collator,
        compute_metrics=compute_metric,
        preprocess_logits_for_metrics=preprocess_logits_for_metrics,
        formatting_func=prepare_sample_text,
    )
2024-04-22T07:41:27Z
I am training the semantic segmentation model (Mask2Former) and hitting exactly this problem.preprocess_logits_for_metricshelps a bit, but doesn’t solve it. Overall, having to predict thewholedataset prior to calculating the metric seems a strange choice to me…