Are dynamic padding and smart batching in the library?
my code:return tokenizer(list(dataset['sentense']),
                          padding = True,
                          truncation = True,
                          max_length = 128 )training_args = TrainingArguments(
    output_dir='./results',          # output directory
    save_total_limit=5,              # number of total save model.
    save_steps=5000,                 # model saving step.
    num_train_epochs=20,              # total number of training epochs
    learning_rate=5e-5,               # learning_rate
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=16,   # batch size for evaluation
...Hello, I understand the docs like thatIf I want to use dynamic padding →padding=Trueor not (max_length) →padding = 'max_lengthIs it right?And I want to use smart batching,Doesper_device_train_batch_sizeautomatically support this feature?If not, I wonder if there is anything that make me to use smart batching.Thanks!!
2021-09-30T09:42:14Z
Hi,This video makes it quite clear:What is dynamic padding? - YouTubeIn order to use dynamic padding in combination with the Trainer, one typically postpones the padding, by only specifyingtruncation=Truewhen preprocessing the dataset, and then using theDataCollatorWithPaddingwhen defining the data loaders, which will dynamically pad the batches.
2021-09-30T14:45:25Z
minji:And I want to use smart batching,Doesper_device_train_batch_sizeautomatically support this feature?If not, I wonder if there is anything that make me to use smart batching.Thanks!!@nielsrThanks. what about smart batching? Is there any tutorial video??
2021-09-30T16:07:09Z
If by smart batching you mean grouping together samples of the same length, it is implemented in theTrainerby addinggroup_by_length=Truein yourTrainingArguments.
2021-11-29T03:52:32Z
Just wondering - how would you usually do this without using the Trainer?
2021-12-09T01:22:37Z
@sguggerhiiif i usedgroup_by_length=Truein myTrainingArguments.this mean applying smart batch??? please answer me
2021-12-09T01:29:41Z
Please do not post the same message three times and tag users agressively like you did. You can always edit your message instead of reposting the same thing.
2021-12-09T01:58:28Z
@sguggersorry i think no one hear me as the other websiteplz can answer me about my question
2021-12-10T05:22:23Z
I understand like it
2022-06-20T22:38:53Z
What if we get this error?Using pad_token, but it is not set yet.Seems that we must also specify what to pad with. It is not the default 0 I presume. Thanks!
2022-06-21T05:45:47Z
I think you need to set a pad token like thistokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
2022-06-21T14:05:28Z
Yes, I did that thank you. Byeos_token, does that just add the last token in line to fill the spots to reach uniformity across the batch? So its value depends on the last token it sees.
2023-03-01T18:32:52Z
@sguggerThis is completely off topic but do you think we could implement grouping by length inside a pipeline to prevent slowdowns due to large differences in sequence lengths? This would only be implemented for users that run thepipelineon aDatasetobject. I’d be happy to contribute this. What would be an appropriate forum to discuss details?
2023-03-01T21:31:26Z
Might be an interesting idea cc@Narsil
2023-04-07T12:07:10Z
Streamline the discussion on dynamic/smart batching,are there existing libraries that does that well with native Pytorch/Tensorflow?(Out of curiosity) Also asking onpytorch - Dynamic batching and padding batches for NLP in deep learning libraries - Data Science Stack Exchange