Fine-Tune BART using "Fine-Tuning Custom Datasets" doc
I am trying to fine-tune BART for a summarization task using the code on the “Fine Tuning with Custom Dataset” page (https://huggingface.co/transformers/custom_datasets.html). The data is a subset of the CNN/Daily Mail data.I am encountering two different errors. The first comes when I implement the exact code on the page:"TypeError: new(): invalid data type ‘str’I assume this is caused by the fact that the labels are not encoded/tokenized, instead they are strings.  I see this is the case in the sample code on the page, the labels are not tokenized.If I tokenize the labels and run the code I receive a different error:“Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers”I am not sure how to interpret that.  I get the same errors whether I am using the Trainer class or using native PyTorch.  Any suggestions?  Thanks!
2020-10-19T15:07:23Z
It’s a bit hard to help you without seeing the code you run.
2020-10-19T15:15:39Z
Hi@Buckeyes2019,Not a direct answer to your question, but you can use the scripts inexamples/seq2seqhere(finetune.pyorfinetune_trainer.py)  for fine-tuning BART and other s2s models. It supports custom datasets as well. All you’ll need to do is get the data in the required format mentioned in the redme.
2020-10-19T16:54:39Z
Sorry, here is the code that produces the error:“Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers”<—>from sklearn.model_selection import train_test_splitfrom transformers import BartTokenizertokenizer = BartTokenizer.from_pretrained(‘facebook/bart-large-cnn’)train_texts, val_texts, train_labels, val_labels = train_test_split(df.articles, df.highlights, test_size=.2)train_texts = train_texts.values.tolist()train_labels = train_labels.values.tolist()val_texts = val_texts.values.tolist()val_labels = val_labels.values.tolist()train_encodings = tokenizer(train_texts, truncation=True, padding=True)train_label_encodings = tokenizer(train_labels, truncation=True, padding=True)val_encodings = tokenizer(val_texts, truncation=True, padding=True)val_label_encodings = tokenizer(val_labels, truncation=True, padding=True)import torchclass PyTorchDatasetCreate(torch.utils.data.Dataset):definit(self, encodings, labels):self.encodings = encodingsself.labels = labelsdef __getitem__(self, idx):
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    item['labels'] = torch.tensor(self.labels[idx])
    return item

def __len__(self):
    return len(self.labels)train_dataset = PyTorchDatasetCreate(train_encodings, train_label_encodings)val_dataset = PyTorchDatasetCreate(val_encodings, val_label_encodings)from transformers import BartForConditionalGeneration, Trainer, TrainingArgumentstraining_args = TrainingArguments(output_dir=’./results’,          # output directorynum_train_epochs=3,              # total number of training epochsper_device_train_batch_size=1,  # batch size per device during trainingper_device_eval_batch_size=1,   # batch size for evaluationwarmup_steps=200,                # number of warmup steps for learning rate schedulerweight_decay=0.01,               # strength of weight decaylogging_dir=’./logs’,            # directory for storing logslogging_steps=10)model = BartForConditionalGeneration.from_pretrained(“sshleifer/distilbart-cnn-12-6”)trainer = Trainer(model=model,                         # the instantiatedTransformers model to be trainedargs=training_args,                  # training arguments, defined abovetrain_dataset=train_dataset,         # training dataseteval_dataset=val_dataset)             # evaluation datasettrainer.train()<---->The code for the other error ("TypeError: new(): invalid data type ‘str’) is identical except the labels (train and val) are not tokenized.  Appreciate your help!
2020-10-26T13:20:26Z
Is the correct format the following:6 filestrain.sourcetrain.targetval.sourceval.targettest.sourcetest.targetEach with one text per line in the .source file and the corresponding summarization in the same line in the .target file?Is there a better documentation what else need to be changed (e.g. data_dir or output_dir)?In the readme it only states: “you need to specify data_dir, output_dir and 	model_name_or_path”.Or does anyone have a public respo where they finetuned one model?
2020-10-26T14:57:55Z
I was able to get this to work:https://ohmeow.com/posts/2020/05/23/text-generation-with-blurr.html
2020-10-28T10:33:57Z
Thanks for the link! But how can you use a custom dataset on that apporach?They use adataset. In the official documentation they are using a single csv file for training:pd.read_csv(path/'cnndm_sample.csv'), but shouldn’t you use at least train and validation?Edit:The answer is:dblock = DataBlock(blocks=blocks,                     get_x=ColReader('article'),                     get_y=ColReader('highlights'),                     splitter=RandomSplitter())They split the single csv file into two dataloader object.