Long wait time between evaluate and save (checkpoint creation)
I’m experimenting with whisper fine-tuning and encounter unreasonable long wait times after the evaluation phase finishes and the checkpoint gets generated.During this period, I don’t see any reasonable work in Windows Task Manager, i.e. no GPU cuda/copy works, CPU ~13%, and no disk activity (only the core where the main thread is running has a bit higher usage).In the following example, I use an augmented Common Voice language set with eval_steps=save_steps=600. I’m using Trainer.train() on GPU.Here are the durations on i7-8900K & rtx-3090 for that 600 stepsTrain Phase: 0:21:43Eval Phase: 0:08:28Extra Wait : 0:13:15After the waiting part, the process continues. I’m using the following if applicable:per_device_train_batch_size = 64,
    gradient_accumulation_steps = 1,
    per_device_eval_batch_size = 16,
    eval_accumulation_steps = 1,

    optim = "adamw_torch",
    tf32 = True,
    fp16 = True,

    gradient_checkpointing = True,
    predict_with_generate = True,

...etc...Where should I look? What can be the culprit?
2023-05-30T19:35:11Z
Here is the CPU usage during thewaitperiod, except the Python code, I have browsers and VS Code open.
2024-09-02T14:01:19Z
Having same problem. Did you solve in some way?Tnx a lot in advance.
2024-09-02T21:33:51Z
No, I lived with it. But I did not train something for a while, so newer versions can be better - I hope.When posting this, I forgot that period is the backpropagation duration, which seems to be single core (I think it is at the last logical core). I wouldn’t expected that to be so long though.Just a guess, without looking into details…
2024-09-02T23:32:02Z
It did not get better I am affraid.I opened another topic. Hoping someone more experienced will tell
2024-09-03T16:35:33Z
I solved by implementing training with pytorch lightning.This example push model to hub but you can easily get a .ckpt classic torch format to reload.github.comNielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "72998908b7844f009be2058a601f5a8e": {This file has been truncated.show original
2024-09-03T18:12:57Z
@FDM1, wrong thread?
2024-09-04T15:07:50Z
Nono. If you use pytorch lighting for training it is faster in both training and saving model.Link is example implementation for custom dataset.
2024-09-04T17:52:02Z
Oh, never used it, I’ll look at it. Thank you for sharing.
2024-09-16T14:08:57Z
Hi@FDM1, can you provide a resource I can use to learn how to switch to it?I am trying to finetune whisper and saving checkpoints takes 2 hours!!!How do I implement this?Thanks in advance.