Purpose of padding and truncating
I have read thePreprocessing Data page. I understand what padding and truncating are doing, but I’m not sure I understand the reason for doing either of them. Can anyone help me understand the purpose for doing them? Thanks in advance!
2020-07-21T15:24:28Z
Hi@aclifton314,padding :Padding is used to make all examples same length so that you can pack them in batch, sequences with uneven length can’t be batched. So if a sequence is shorter, than your max length then padding is used to make that sequence longer. Also some model might expect fixed length input, so padding help there too.truncation:Most of the models have max_lengths defined for them (there are exceptions, model with relative attention can take arbitrarily long sequences) for ex.for BERT max_length is 512, so if one of your sequence is longer than that you can’t feed it directly, so you need to truncate (drop extra tokens) to make the sequence smaller.Hope this helps
2020-07-21T15:27:11Z
@valhallacrystal clear. Thank you very much!
2020-08-03T12:23:42Z
So if I understand right, the sentence that is too long will see some of its tokens deleted, without using these extra tokens to train the future model?Does these tokens are randomly selected or if the max_lenght=100 then tokens > 100 will be deleted?Thank you
2020-08-03T12:44:48Z
Hi@kasar3, if you enabletruncationthen the extra tokens will be deleted , they are not selected randomly.  Thisguideexplains padding and truncation in detail.
2020-08-03T13:25:41Z
Alright thank you.I have an other question, don’t know of it’s the right place… For example, in the case of text classification using BERT model (512 tokens max): let’s say that I have a sentence of 1000 tokens, what should I do to tokenize it like BERT want and still being able to classife it later. Should I split it manually in sentences of length <512 ?For exemple split it in 2 sentences of 500 tokens while duplicate the text classification label of the original sentence?
2020-08-03T16:50:24Z
I don’t have a good answer for this, but I have seen people breaking longer documents into chunks, getting the label for each chunk and then aggregate it.Second approach is, now with recent models like longformer it’s possible to train models with large sequence length so that you won’t need to chunk the document.@sguggermight have a better answer.
2020-08-03T17:01:43Z
More than the labels, I’d get the pooler output of each chunk and then average them before using the classifier. This will require you to change the code ofBertForSequenceClassificationa little bit.