How to train with very long sequences?
Hi, I’m trying to fine-tune a classifier on very long sequences that can reach up to 6000000 characters and even more. I started with GPT-2 model, and had to truncate the sequences to the limit of the model, which is 1024 if I understood correctly.Is there a way for me to train this or a different model on longer sequences?
2022-05-18T20:12:43Z
There is longformer, a transformer created for long sequences. I think in their paper they claim to be able to attend tens of thousands of tokens.
2022-05-20T06:52:02Z
thank you!