Data collator for training bart from scratch
Hello,I would like to train bart from scratch.It seems the official example script is not available yet (if any, please tell me!).So I try to have one by modifying the example scriptsrun_mlm.pyandrun_clm.py.And not sure how to set the data collator part for bart.Inrun_mlm.py, DataCollatorForLanguageModeling is used:github.comhuggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_mlm.py#L342# https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.maptokenized_datasets = tokenized_datasets.map(group_texts,batched=True,num_proc=data_args.preprocessing_num_workers,load_from_cache_file=not data_args.overwrite_cache,)# Data collator# This one will take care of randomly masking the tokens.data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=data_args.mlm_probability)# Initialize our Trainertrainer = Trainer(model=model,args=training_args,train_dataset=tokenized_datasets["train"] if training_args.do_train else None,eval_dataset=tokenized_datasets["validation"] if training_args.do_eval else None,tokenizer=tokenizer,data_collator=data_collator,)Inrun_clm.py, default_data_collator is used:github.comhuggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_clm.py#L310)# Initialize our Trainertrainer = Trainer(model=model,args=training_args,train_dataset=lm_datasets["train"] if training_args.do_train else None,eval_dataset=lm_datasets["validation"] if training_args.do_eval else None,tokenizer=tokenizer,# Data collator will default to DataCollatorWithPadding, so we change it.data_collator=default_data_collator,)# Trainingif training_args.do_train:trainer.train(model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None)trainer.save_model()  # Saves the tokenizer too for easy upload# EvaluationCan someone give some advice about how to set the data collator for bart?Thanks.
2020-12-06T13:19:58Z
@zuujhytMaybe try creating differentDatasetclasses using differentDataCollators. You could then use PyTorch Lightning to create aDataloaderw/ multipleDatasetclasses likethis, and train Bart on that. I’ll try doing this.Let me know if you have any updates on this.