Getting error while resuming the training with a single GPU
Hello all,I’m new to the Huggingface library though I’m familiar with the deep learning in general and some of the DL libraries.so I’m trying to train a BERT model from scratch with a custom text. I followed the instructionshereand managed to get the training going with 2 GPUs. However, later on, I had to stop the training and resume with a single GPU. Then, I got the error below. I looked up this forum, StackOverflow etc. but couldn’t find any solution so far.The environment info is as follows (the result of the commandtransformers-cli env):transformers version: 4.10.0.dev0Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.10Python version: 3.8.5PyTorch version (GPU?): 1.8.1+cu111 (True)Tensorflow version (GPU?): not installed (NA)Flax version (CPU?/GPU?/TPU?): not installed (NA)Jax version: not installedJaxLib version: not installedUsing GPU in script?: YesUsing distributed or parallel set-up in script?: First yes, then no (I guess)I created a big text file where I got a sentence at each line and a newline in between documents (I guess it’s called the NSP format) and I’m using a tokenizer from the Huggingface repo.So I started the training with this command:python run_mlm.py  --model_type bert  --train_file <path_to_my_train_file> --validation_file <path_to_my_val_file>  --do_train  --do_eval  --output_dir /my/local/path/test-mlm --tokenizer_name dbmdz/bert-base-turkish-cased --cache_dir /my/local/cache/dir --line_by_line --save_total_limit 2 --logging_dir /my/log/dirI have two GPUs and this command was using both of them as intended. Then, when I stopped the training and wanted to resume with one GPU with the following command (Basically I just addedCUDA_VISIBLE_DEVICES=1at the beginning):CUDA_VISIBLE_DEVICES=1 python run_mlm.py  --model_type bert  --train_file <path_to_my_train_file> --validation_file <path_to_my_val_file>  --do_train  --do_eval  --output_dir /my/local/path/test-mlm --tokenizer_name dbmdz/bert-base-turkish-cased --cache_dir /my/local/cache/dir --line_by_line --save_total_limit 2 --logging_dir /my/log/dirFirst, it skipped some batches to resume from the last checkpoint and threw this error trace while loading it:Traceback (most recent call last):File “run_mlm.py”, line 550, inmain()File “run_mlm.py”, line 501, in maintrain_result = trainer.train(resume_from_checkpoint=checkpoint)File “/my/local/path/venv/lib/python3.8/site-packages/transformers/trainer.py”, line 1262, in trainself._load_rng_state(resume_from_checkpoint)File “/my/local/path/venv/lib/python3.8/site-packages/transformers/trainer.py”, line 1477, in _load_rng_statetorch.cuda.random.set_rng_state_all(checkpoint_rng_state[“cuda”])File “/my/local/path/venv/lib/python3.8/site-packages/torch/cuda/random.py”, line 73, in set_rng_state_allset_rng_state(state, i)File “/my/local/path/venv/lib/python3.8/site-packages/torch/cuda/random.py”, line 64, in set_rng_state_lazy_call(cb)File “/my/local/path/venv/lib/python3.8/site-packages/torch/cuda/__init__.py”, line 114, in _lazy_callcallable()File “/my/local/path/venv/lib/python3.8/site-packages/torch/cuda/random.py”, line 61, in cbdefault_generator = torch.cuda.default_generators[idx]IndexError: tuple index out of rangeIt works with no problem if I removeCUDA_VISIBLE_DEVICES=1but then it uses both of the GPUs. I had a desperate attempt of doublingper_device_train_batch_size(default is 8) hoping it would balance out the single GPU index error:CUDA_VISIBLE_DEVICES=1 python run_mlm.py  --model_type bert  --train_file <path_to_my_train_file> --validation_file <path_to_my_val_file>  --do_train  --do_eval  --output_dir /my/local/path/test-mlm --tokenizer_name dbmdz/bert-base-turkish-cased --cache_dir /my/local/cache/dir --line_by_line --save_total_limit 2 --logging_dir /my/log/dir  --per_device_train_batch_size 16This didn’t work unfortunately. So I got stuck at this point.Any help is appreciated. Thanks!
2024-06-13T13:59:44Z
Similar issue.I am using DPP using torchrunEarlier I am training a model on 8 GPUs with DPP, saved a checkpoint, with following files in it,rng_state_0.pthrng_state_1.pthrng_state_2.pthrng_state_3.pthrng_state_4.pthrng_state_5.pthrng_state_6.pthrng_state_7.pthNow I want to run the same script model training from the checkpoint but with 6 GPUs only.I am facing following error:IndexError: tuple index out of range
    callable()
  File "site-packages/torch/cuda/random.py", line 61, in cb
    default_generator = torch.cuda.default_generators[idx]
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^How can I resume my training from the checkpoint with lesser GPU environment with DPP ?