Dataset expected by Trainer
Hello everyone,I am rewriting some old code to use the new tokenizer syntax and the Trainer class but I believe I am missing something.This is how I am building the training dataset to be passed to the Trainer constructor:encoded_texts = tokenizer(texts, padding = True, truncation = True, return_tensors = 'pt')
    labels = torch.tensor(labels)
    dataset = TensorDataset(encoded_texts['input_ids'], encoded_texts['attention_mask'], labels)Can you please help me understand what I am doing wrong/missing? When I runtrainer.train()I get the following error:vars() argument must have __dict__ attributeThanks in advance!
2020-07-09T15:54:03Z
I’ll give you the full picture.The workflow:You create an instance ofGlueDataset(data_args, tokenizer). Then you pass it toTrainer(...)class. In trainer, you also pass indefault_data_collator. The reason is thatGlueDatasetreturnInputExamplewhich is HF specific and cannot be used by Pytorch dataloader directly. So thedefault_data_collatortakes inList[InputExamples]and returns a dict. This dict is then used by the dataloader.So in trainer, if you passdefault_data_collatorwithTensorDataset, it won’t work directly (That’s why you’re getting the error). This error is raised when dataloader will pass the batch todefault_data_collator. I’d suggest using the default Pytorchcollate_fnwith yourTensorDataset, it would work just fine.One more additional thing:Make sure the dataloader returns the dict with same key valuesforwardmethod expects.Inside_training_step, you’ll passinputsto the function, and then after the inputs are passed kept on gpu, the function does:output = model(**inputs)In this case, the keyword arguments have to match. In case, they don’t,  you can inherit fromTrainerand redefine your own method.I hope this answers your question.
2020-07-10T16:28:11Z
Thank you very much for your help!I tried different approaches; currently I am testing a solution that involves writing my owndata_collatorto pass to theTrainerinstance:def dummy_data_collector(features):
    batch = {}
    batch['input_ids'] = torch.stack([f[0] for f in features])
    batch['attention_mask'] = torch.stack([f[1] for f in features])
    batch['labels'] = torch.stack([f[2] for f in features])
    
    return batchThis seems to be working, but I will need to do some more testing
2020-07-11T09:02:42Z
I’m glad it worked out.
2020-09-28T08:56:49Z
I am not sure I understand how the Trainer class identifies the target from the features.  In the given example in the documentation, Trainer is passed “train_dataset” with no X and Y specified. Is this already assigned somewhere on the dataset object?from transformers import BartForSequenceClassification, Trainer, TrainingArgumentsmodel = BartForSequenceClassification.from_pretrained(“facebook/bart-large-cnn”)training_args = TrainingArguments(output_dir=’./results’,          # output directorynum_train_epochs=3,              # total # of training epochsper_device_train_batch_size=2,  # batch size per device during trainingper_device_eval_batch_size=2,   # batch size for evaluationwarmup_steps=100,                # number of warmup steps for learning rate schedulerweight_decay=0.01,               # strength of weight decaylogging_dir=’./logs’,            # directory for storing logs)trainer = Trainer(model=model,                         # the instantiatedTransformers model to be trainedargs=training_args,                  # training arguments, defined abovetrain_dataset=train_df,         # training dataset****    eval_dataset=eval_df)**
2020-09-28T14:32:54Z
AllTransformers models will return the loss when fed with the inputs and labels (usually namedlabels). TheTrainerthus expects each element of the dataset you pass to be a dictionary with all the inputs the model expects to return the loss (including thoselabels).