[Beginner] fine-tune Bart with custom dataset in other language?
@valhalla@sshleiferHi, I’m new to the seq2seq model. And I want to fine-tune Bart/T5 for the summarization task. There are some documents related to the fine-tuning procedure.Such ashttps://github.com/huggingface/transformers/tree/master/examples/seq2seqhttps://ohmeow.github.io/blurr/modeling-seq2seq-summarization/And also thanks for the distilbart version.But my custom dataset is in Japanese. Directly fine-tuning might be impossible. Is it necessary to train a new bpe tokenizer with Japanese data? But I don’t know how to do it.The second way is to use an existing Japanese tokenizer likebert-japanese, but could I just use it for Bart? How to modify it?The third way is to use a multilingual model like MBart or MT5. I haven’t tested it. Could I just fine-tune them with the Japanese dataset ?Please forgive me if this is a stupid question. Thanks in advance.Thanks in advance.
2021-01-22T06:30:09Z
Hi@HeroadZBart is trained in English so I don’t think fine-tuning it directly will help.  If you want to train a model from scratch in a new language then yes, you should train a new tokenizer. To train a new tokenizer checkout thetokenizerslibrary.And both MBart and MT5 support Japanese so that would be a good starting point.Another option is to leverage language-specific encoder only bert model (in your casebert-japanese) to create a seq2seq model using theEncoderDecoderframework. See this notebook to know more aboutEncoderDecodermodelsLeverage BERT for Encoder-Decoder Summarization on CNN/Dailymail
2021-01-22T07:01:24Z
Thanks for the quick reply!The last bert2bert is amazing!I will try these methods. Thank you very much.