Longformer for text summarization
Hello!Does anyone know how to summarize long documents/news articles using the Longformer library? I am aware that using T5, the token limit is 512.I would really appreciate any help in this area! Thank you
2020-07-26T08:55:51Z
Hi, it’s possible to use Longformer for summerization, the way its done now, is taking BART model and then replacing it’s self attention with longformer sliding window attention so that it can take longer sequences. Check this two issues,first,second, and thisbranchof longformer repo
2020-07-26T10:42:57Z
Hi, I followed the example from thebranchof longformer repo, but it seems that the final output is a tensor instead of words/text. How can I convert it into words?
2020-07-26T11:29:28Z
Note that, these models are not yet fine-tuned for long summarization, you’ll need to fine-tune them yourself or wait till someone does that. And yes, the model returns a tensor, to generate text you’ll need to use thegeneratemethod.Here’s niceblog poston the generate method
2021-01-22T11:02:34Z
the links are broken
2021-11-30T08:59:14Z
Hey@valhalla. Hope you’re well. In your earlier comment here you mention that Longformer for summarisation takes the BART model and replaces it’s self attention. I was under the impression thatthis modelwas based off ROBERTA. Can you confirm if there is a long former model based off BART and if so where it is on the hub?
2021-11-30T09:10:29Z
Hi,LongFormer is an encoder-only Transformer (similar to BERT/RoBERTa), it only has a different attention mechanism, allowing it to be used on longer sequences.The author also released LED (LongFormer Encoder Decoder), which is a seq2seq model (like BART, T5) but with LongFormer as encoder, hence allowing it to be used to summarize long documents for instance, or translate long texts.Weights are on the hub:Models - Hugging Face
2021-11-30T09:45:12Z
Amazing that’s very helpful thank you. I can see on that link that Allen AI’s LED model is based of bart-base which is ideal. If I were to look to try and convert bart-large to a LED would thisnotebookstill be the right approach or is this for encoder-only models?
2021-11-30T10:55:16Z
kmfoda:notebookI don’t have access to this notebook.
2021-11-30T11:59:07Z
okay no problem. Thanks for your help!
2022-08-06T16:54:15Z
Any updates? Can someone share the relevant code snippet from that private notebook publicly?