Setting specific device for Trainer
Hi  I’m trying to fine-tune model with Trainer in transformers,Well, I want to use a specific number of GPU in my server.My server has two GPUs,(index 0, index 1) and I want to train my model with GPU index 1.I’ve read the Trainer and TrainingArguments documents, and I’ve tried the CUDA_VISIBLE_DEVICES thing already. but it didn’t worked for me.import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"(I did it within Jupyter, before I import all libraries)It gave me a runtime error when the trainer tries to initiate self.model = model.to(args.device) line.and the error says like RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable.trainer = Trainer(
 model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset)123944×820 127 KBI’ve also tried torch.cuda.set_device(1), it also didn’t work.I don’t know how to set it up. It seems like I don’t have any options in argument of classPlease help me to handle this problem.Thank you.
2020-08-20T12:37:18Z
Iftorch.cuda_set_device(1)doesn’t work, the problem is in your install. Does the commandnvidia-smishow up two GPUs?
2020-08-21T02:08:36Z
Yes.My code is heretorch.cuda.set_device(1)
 torch.cuda.current_device()
 1

epochs = 3
training_args = TrainingArguments(
do_predict=True,
output_dir=f'./results',
overwrite_output_dir=True,
do_train=True,
num_train_epochs=epochs,
per_device_train_batch_size=190,
logging_steps=10,
learning_rate=5e-05,
warmup_steps=500, 
save_total_limit = 100,
logging_dir='./logs',
save_steps=50)

training_args.deviceout isdevice(type='cuda', index=0)andtrainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset)

!nvidia-smi2233851×697 89.3 KBI can see two GPUsand nvidia-smi shows that the process is running with GPU#0still.It seems that torch.cuda.set_device(1) doens’t work at all.
2020-08-21T13:10:35Z
Mmmm, looking at the code, I can see that Trainer ignores the default device indeed. Will fix that soon.
2020-08-22T03:03:14Z
I got a discussion with my mate, He suggested to reinstall GPU driver or restart the server.I’ll try it and tell what happens to you.Thanks a lot.
2021-05-06T05:38:44Z
Is it implemented?, I could not figure out how to set device one for training i tried above methods.
2022-10-25T05:49:47Z
Hi I am experiencing the same issue when using setfit is there any solution?
2022-11-29T20:12:41Z
Hi@sgugger,Did you fix this bug?Today I faced with the same issue when I set visible GPU to 1, but GPU 0 is still usedimport os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"Note: I’m using the latest transformers module in python 3.7
2023-03-01T13:10:06Z
ithieund:GPU to 1, but GPU 0 is still usedI am having similar issue. Did you manage to get it work?
2023-04-10T12:34:38Z
Same issue. Why when loading a model using .from_pretrained we can specify device_map, but we can’t specify device_map when defining a Trainer?
2023-05-18T21:28:03Z
This issue still exists. Thanks.
2023-05-25T07:34:07Z
Meanwhile, this can do a trick:trainer = Trainer(
    model = model,
    .
    .
    device=0 # id of the device to use (int) -> for GPU(s): 0, 1 or for CPU: -1
)
2023-06-14T18:12:04Z
sorry to bother you, but what do you mean here?Trainer doesn’t have a parameterdevice
2023-06-15T16:13:59Z
Is there any solution to this?Still can’t get Trainer to use a particular device in transformersv4.30.2even after setting environment variableimport os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]= "2"It still takes GPU device 0[In]: trainer.args.device
[Out] :device(type='cuda', index=0)
2023-07-10T04:19:57Z
The same issue still exist, literally dealing with this right now
2023-07-21T08:48:38Z
Really? I just didimport torch
torch.cuda.set_device(1)and it works flawlessly. Doesn’t fix that the models are pretty big relative to my GPU memory and I have to use quite small batch sizeUsingtorch==2.0.1and SetFit library.
2023-07-30T01:47:32Z
It still doesn’t work. Any updates? Thanks!
2023-08-31T13:11:58Z
pyNVMLis kinda terrible but if you can’t access the GPU usingpyNVMLthen the problem is with your Python / Jupyter, nottorchor any of the libraries that build on it.!nvidia-smiis not enough to verify that since the command is just sent directly to bash.Here is the website.PyPInvidia-ml-pyPython Bindings for the NVIDIA Management LibraryEDIT: But it sounds like the GPU is busy. Are you sure you don’t have another Jupyter session running? Even if it is done training it won’t release the GPU until you shut down the kernel.
2023-10-05T21:07:44Z
A solution that works for me:import osos.environ[“CUDA_DEVICE_ORDER”]=“PCI_BUS_ID”os.environ[“CUDA_VISIBLE_DEVICES”]=“0”But import this FIRST, before anything from the pytorch or transformer libraries.
2023-10-13T07:19:43Z
Thanks@josejames00, your solution works to me.
No date available
No reply text available