Sending a Dataset or DatasetDict to a GPU
Hi, relatively new user of Huggingface here, trying to do multi-label classfication, and basing my code offthisexample.I have put my own data into a DatasetDict format as follows:df2 = df[['text_column', 'answer1', 'answer2']].head(1000)
df2['text_column'] = df2['text_column'].astype(str)
dataset = Dataset.from_pandas(df2)

# train/test/validation split
train_testvalid = dataset.train_test_split(test_size=0.1)
test_valid = train_testvalid["test"].train_test_split(test_size=0.5)

# put into DatasetDict
datasets = DatasetDict({
    "train": train_testvalid["train"],
    "test": test_valid["test"],
    "valid": test_valid["train"]})Later, I load the model usingmodel = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", 
                                                           problem_type="multi_label_classification", 
                                                           num_labels=len(labels),
                                                           id2label=id2label,
                                                           label2id=label2id).to(device)and then check that the model is on GPU usingnext(model.parameters()).is_cuda; if I comment out.to(device), the model is not sent to GPU.My problem comes when it’s time to train the model as follows:trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"].to(device),
    eval_dataset=encoded_dataset["valid"].to(device),
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()which causes the error:AttributeError: 'Dataset' object has no attribute 'to'But if Idon’ttry and send the train and eval datasets to GPU, I get the error:RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)So my question is: how can I send these datasets to the GPU - where the model already is - in order to efficiently train and validate the model using them?Thank you!
2022-04-26T12:22:35Z
By default, the Trainer will use the GPU if it is available. It will automatically put the model on te GPU as well as each batch as soon as that’s necessary. So just remove all.to()calls that you made manually.github.comhuggingface/transformers/blob/8afaaa26f5754948f4ddf8f31d70d0293488a897/src/transformers/training_args.py#L1088device = torch.device("cuda", self.local_rank)self._n_gpu = 1elif self.local_rank == -1:# if n_gpu is > 1 we'll use nn.DataParallel.# If you only want to use a specific subset of GPUs use `CUDA_VISIBLE_DEVICES=0`# Explicitly set CUDA to the first (index 0) CUDA device, otherwise `set_device` will# trigger an error that a device index is missing. Index 0 takes into account the# GPUs available in the environment, so `CUDA_VISIBLE_DEVICES=1,2` with `cuda:0`# will use the first GPU in that env, i.e. GPU#1device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")# Sometimes the line in the postinit has not been run before we end up here, so just checking we're not at# the default value.self._n_gpu = torch.cuda.device_count()else:# Here, we'll use torch.distributed.# Initializes the distributed backend which will take care of synchronizing nodes/GPUsif not torch.distributed.is_initialized():torch.distributed.init_process_group(backend="nccl")device = torch.device("cuda", self.local_rank)self._n_gpu = 1
2022-04-26T12:34:27Z
Hi! As@BramVanroypointed out, ourTrainerclass uses GPUs by default (if they are available from PyTorch), so you don’t need to manually send the model to GPU. And to fix the issue with the datasets, set their format totorchwith.with_format("torch")to return PyTorch tensors when indexed.
2022-04-26T13:18:53Z
Thanks you both for responding so quickly.I can confirm that GPU is available usingtorch.cuda.is_available(), and I have also done.set_format("torch")on the Datasets. I have removed any explicit.to()calls.However, if I remove the  explicit.to()call on the model, then the model is no longer on the GPU according tonext(model.parameters()).is_cuda→ returns False.More importantly, I also still get the Runtime Error:Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!I was trying to post a minimal example above, but I nowsuspect that the problem is  that some of the encoding is done on the GPU perhaps? So here is a larger section of code - apologies in advance if this is excessive, but not sure which part is causing the error.from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def preprocess_data(examples):
  # take a batch of texts
  text = examples["answer_no_tags"]
  # encode them
  encoding = tokenizer(text, padding="max_length", truncation=True, max_length=128)
  # add labels
  labels_batch = {k: examples[k] for k in examples.keys() if k in labels}
  # create numpy array of shape (batch_size, num_labels)
  labels_matrix = np.zeros((len(text), len(labels)))
  # fill numpy array
  for idx, label in enumerate(labels):
    labels_matrix[:, idx] = labels_batch[label]

  encoding["labels"] = labels_matrix.tolist()
  
  return encoding

encoded_dataset = datasets.map(preprocess_data, batched=True, 
                              remove_columns=dataset.column_names)Loading modelfrom transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", 
                                                           problem_type="multi_label_classification", 
                                                           num_labels=len(labels),
                                                           id2label=id2label,
                                                           label2id=label2id)

from transformers import TrainingArguments, Trainer

args = TrainingArguments(
    f"bert-finetuned-sem_eval-english",
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=5,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model=metric_name,
    #push_to_hub=True,
)

from sklearn.metrics import f1_score, roc_auc_score, accuracy_score
from transformers import EvalPrediction
import torch
    
# source: https://jesusleal.io/2021/04/21/Longformer-multilabel-classification/
def multi_label_metrics(predictions, labels, threshold=0.5):
    # first, apply sigmoid on predictions which are of shape (batch_size, num_labels)
    sigmoid = torch.nn.Sigmoid()
    probs = sigmoid(torch.Tensor(predictions))
    # next, use threshold to turn them into integer predictions
    y_pred = np.zeros(probs.shape)
    y_pred[np.where(probs >= threshold)] = 1
    # finally, compute metrics
    y_true = labels
    f1_micro_average = f1_score(y_true=y_true, y_pred=y_pred, average='micro')
    roc_auc = roc_auc_score(y_true, y_pred, average = 'micro')
    accuracy = accuracy_score(y_true, y_pred)
    # return as dictionary
    metrics = {'f1': f1_micro_average,
               'roc_auc': roc_auc,
               'accuracy': accuracy}
    return metrics

def compute_metrics(p: EvalPrediction):
    preds = p.predictions[0] if isinstance(p.predictions, 
            tuple) else p.predictions
    result = multi_label_metrics(
        predictions=preds, 
        labels=p.label_ids)
    return resultFinally, training:trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset["valid"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()Which returns:RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper__index_select)
2022-04-28T13:05:01Z
Is this the entire code? I can’t find the part where you change the dataset format totorch.
2022-04-28T13:30:56Z
The model will be moved to the GPU after you initialize the trainer - not before that.github.comhuggingface/transformers/blob/dced262409177586bb510b6b724c762fb89da0e8/src/transformers/trainer.py#L381-L382if self.place_model_on_device:self._move_model_to_device(model, args.device)You can verify that the trainer will make use of the GPU by checkingtrainer.args.device. If that is a GPU, then everything the trainer does will correctly use the GPU.What I suspect instead is that there is a discrepancy between devices in your custommulti_label_metricsfunction, which the trainer of course does not control. Check whetherpredictionsandlabelsare on the same device.
2022-04-28T13:52:34Z
Oh, I think this is a Transformers bug (seeWhen running the Trainer cell, it found two devices (cuda:0 and CPU) · Issue #31 · nlp-with-transformers/notebooks · GitHub). Updating Transformers to the newest version withpip install -U transformersshould fix the issue.
2022-05-03T13:32:13Z
Thanks for your help@BramVanroyand@mariosasko- much appreciated. I updated Transformers and that has fixed the error, so have marked as solution. Cheers.
2022-07-07T02:44:25Z
Also, remember to update the ‘accelerate’ package for making it compatible
2022-08-04T21:30:24Z
Hi@BramVanroy,I’m experiment the use of mac M1 with transformers 4.21.1 and pytorch 1.13.0.dev20220803.Is the “Trainer will use the GPU if it is available” also true in the case of M1 (“mps”)? I’m also having issues with this matter. I can’t train with the M1 GPU, only CPU.Thanks
2022-08-09T15:11:30Z
You can have a look at this issue for more:TrainingArguments does not support `mps` device (Mac M1 GPU) · Issue #17971 · huggingface/transformers · GitHub
2022-08-09T15:27:48Z
Thanks. Already saw it. But even so there are some problems. First I was getting bad results when using mps (this issue issolved).  A second issue is related to performance. I’m not seeing a considerable increase in speed in comparison to the CPU. The issue is also addressedhere.
2023-04-13T09:35:00Z
Before running the Trainer, I want to manually compute the loss:labels = train_data[‘labels’][0]input_ids = train_data[‘input_ids’][0]attention_mask = train_data[‘attention_mask’][0]outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)loss = outputs.lossWhat’s the way to manually convert the dataset to GPU tensor?
2023-04-13T15:03:11Z
Hi! You can dotrain_data.set_format("torch", device="cuda")to send the dataset’s samples to GPU when indexing into the dataset.
2023-10-05T03:32:42Z
HiI understand that if GPU is available the trainer will automatically send the model to GPU, but will the trainer automatically send my tokenized inputs and labels to GPU without having to explicitly include the code.to(device)?My guess is yes but I would still like to double check.I have been trying to fine-tune the nllb translation model and I was wondering if I need to include the code.to(device)in my compute metrics function?metric = evaluate.load("sacrebleu")

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    result = {"bleu": result["score"]}

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    result = {k: round(v, 4) for k, v in result.items()}
    return result
2024-06-01T20:02:11Z
This worked. We have to do this to do perform finetuning in a gpu. without setting data to data.with_format(“torch”) it will be processed in cpu.
2024-07-18T11:37:16Z
Hello,I am trying to fine-tune a model on my M1 Mac using mps, but getTypeError: can't convert mps:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.inside thetransformers/data/data_collator.pyfile.So I instantiate the tokenizer, move the model tomps, load the dataset, move the dataset tompswith set_format (without it I getPlaceholder storage has not been allocated on MPS device!, which is due to data not being moved tomps), then instantiate and run the Trainer:tokenizer = AutoTokenizer.from_pretrained(self.config.model_ckpt)
model_pegasus = AutoModelForSeq2SeqLM.from_pretrained(self.config.model_ckpt).to(torch.device('mps'))
seq2seq_DC = DataCollatorForSeq2Seq(tokenizer, model = model_pegasus)
ds = load_from_disk(path_to_ds)
ds.set_format('torch', device="mps")
trainer = Trainer(
                    model = model_pegasus,
                    args = trainer_args,
                    tokenizer = tokenizer,
                    data_collator=seq2seq_DC,
                    train_dataset=ds['train'],
                    eval_dataset=ds['validation']
                )
trainer.train()Any ideas why this happens, and why the Trainer doesn’t automatically move the inputs to the correct device?