Is Transformers using GPU by default?
I’m instantiating a model with thistokenizer = AutoTokenizer.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")
model = AutoModelForSequenceClassification.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")Then running a for loop to get prediction over 10k sentences on a G4 instance (T4 GPU). GPU usage (averaged by minute) is a flat 0.0%. What is wrong? How to use GPU with Transformers?
2021-07-19T17:56:07Z
This like with every PyTorch model, you need to put it on the GPU, as well as your batches of inputs.
2021-07-20T07:22:55Z
You can take a look at this issueHow to make transformers examples use GPU? · Issue #2704 · huggingface/transformers · GitHubIt includes an example for how to put your model on GPU.device = "cuda:0" if torch.cuda.is_available() else "cpu"
sentence  = 'Hello World!'
tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')

inputs    = tokenizer(sentence, return_tensors="pt").to(device)
model     = model.to(device)
outputs   = model(**inputs)
2022-03-23T11:50:02Z
@sguggerAre there any samples of how Huggingface Transformer finetuning should be done using GPU please?
2022-05-12T23:57:01Z
Hi@sguggerwere you able to figure this out?
2022-12-31T12:33:32Z
I had the same issue - to answer this question, if pytorch + cuda is installed, an e.g. transformers.Trainer class using pytorch will automatically use the cuda (GPU) version without any additional specification.(You can check if pytorch + cuda is installed by checking if thepytorch-cudapackage is installed.)
2023-12-11T01:53:34Z
Is there a way to explicitly disable the trainer from using the GPU? I see something aboutplace_model_on_deviceonTrainerbut it is unclear how to set it toFalse.EDIT: Oh, I see I can setuse_cpuinTrainingArgumentsto False.