Distilbart Truncation
Hello, can anyone please explain to me how is the truncation done for long sequences in Distilbart ?Thank you very much ! This is a very important matter.
2020-10-22T14:54:02Z
max seq len for BART is 1024 tokens, the tokenizer truncates all the tokens beyond that limit
2020-10-22T15:12:56Z
So, the tokens beyond that limit are directly deleted? (Like, there is no special truncation strategy, just a simple deletion)
2020-10-22T15:59:58Z
yes, that’s the default behaviour, but you could also disable truncation and use your own truncation logic.