Training the t5
Hello, on the T5  page it says to train it using justloss = model(input_ids=input_ids, lm_labels=labels).lossThis did nothing but give me a loss value.However when i do thisSummarizationI get the error text, and summary are not recognized.
2022-08-16T05:25:59Z
Could you please show your code and the respective error?
2022-08-16T12:54:07Z
from datasets import load_datasetfrom transformers import T5Tokenizer, T5ForConditionalGenerationfrom transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainerbillsum = load_dataset(“billsum”, split=“ca_test”)billsum = billsum.train_test_split(test_size=0.2)from transformers import AutoTokenizertokenizer = T5Tokenizer.from_pretrained(“google/t5-efficient-xxl-nl4”)model = AutoModelForSeq2SeqLM.from_pretrained(“/media/samuel/New Volume/models/t5-efficient-xxl-nl4”)prefix = "summarize: "def preprocess_function(examples):inputs = [prefix + doc for doc in examples[“text”]]model_inputs = tokenizer(inputs, max_length=1024, truncation=True)with tokenizer.as_target_tokenizer():
    labels = tokenizer(examples["summary"], max_length=128, truncation=True)

model_inputs["labels"] = labels["input_ids"]
return model_inputstokenized_billsum = billsum.map(preprocess_function, batched=True)from transformers import DataCollatorForSeq2Seqdata_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)training_args = Seq2SeqTrainingArguments(output_dir=“./results”,evaluation_strategy=“epoch”,learning_rate=2e-5,per_device_train_batch_size=16,per_device_eval_batch_size=16,weight_decay=0.01,save_total_limit=3,num_train_epochs=1,fp16=True,)trainer = Seq2SeqTrainer(model=model,args=training_args,train_dataset=tokenized_billsum[“train”],eval_dataset=tokenized_billsum[“test”],tokenizer=tokenizer,data_collator=data_collator,)trainer.train()Errior:The following columns in the training set don’t have a corresponding argument inT5ForConditionalGeneration.forwardand have been ignored: title, summary, text. If title, summary, text are not expected byT5ForConditionalGeneration.forward,  you can safely ignore this message.
2022-08-16T13:09:58Z
I believe that error is fine but it doesn’t change my model when i train on this.the training loss is 0.0and the eval loss is nan
2022-08-16T18:14:51Z
I figured it out. you get the loss value from the model and callloss = model(input_ids=input_ids, lm_labels=labels).lossoptimizer.zero_grad()loss.backward()optimizer.step()I just figure it would do this part for you