Finetuning summarization model using long text data
Hello, I want to fine tune pszemraj/led-base-book-summary model on my custom data of Bank Regulatory Document (15-20 pages) but the documents is well above the input token limit I can truncate it but I believe that it will cause a lot of loss of information. Can anyone suggest the right way to fine-tune using long document.
2024-03-12T08:21:10Z
I had similar problem. What I did was to add[EOS]token (or appropriate token for your model) at the end of each document and then when tokenizing the documents used the following setup:trx_text_encoded = tokenizer(
    text = trx_text,
    max_length=tokenizer.model_max_length,
    truncation=True,
    padding="max_length",
    return_overflowing_tokens=True,
    stride=12,
    return_tensors="pt"
)Thereturn_overflowing_tokenswill “chunk” up extra long text into sequences and thestridewill create overlapping tokens at the beginning of sequences.
2024-03-12T13:37:00Z
Thank you so much for your detailed response! Your solution using  the tokenizer setup is incredibly helpful. I’m planning to implement it.I have one more question: could you please elaborate on the significance of adding the [EOS] token at the end of each document? I want to understand its role in the fine-tuning process.
2024-03-13T08:21:30Z
The idea with the[EOS]token was to “tell the model” that there are more than one document as input. So basically, to sort of separate where one document ends and the other begins and all in between is for one document. I thought this could be helpful since the documents are “disjoint” in essence.
2024-03-14T05:00:39Z
Thank you that was incredibly helpful and provided me with valuable insight.
2024-03-20T00:08:39Z
Thanks@itacdonevthis helped me also.Didn’t know about the addtional KWargs like stride, return_overflowing_tokens. Could you please help me , how did you learn about this?I am curating the code snippets like the one you have shared in Transformers Universe space. I want to provide additional context there.Transformers Universe - a Hugging Face Space by Kamaljp
2024-03-20T06:23:50Z
Two main sources that I consulted first:HF documenatationHF Causal lanugage modeling - preparing datasets (YT video)Then I tried the code on a very simple example like couple of sentences withmax_length=4to see it in action and whether it is behaving as expected.Hope this help.