Is IterableDataset automatically reshuffled after each epoch in Trainer?
I’m using a IterableDataset from datasets library and passing it to HF Trainer, something like this:from datasets import load_dataset
from transformers import Trainer, TrainingArguments

ds = load_dataset("my-dataset", streaming=True)

training_args = TrainingArguments(
    output_dir="my_model",
    per_device_train_batch_size=8,
    max_steps=1000,  # Large enough for multiple epochs
)

trainer = Trainer(
    model=my_model,
    train_dataset=ds["train"].shuffle(seed=42),
    eval_dataset=ds["test"],
)
trainer.train()Then, will trainer automatically reshuffle my iterable dataset after each epoch?I cannot find the behavior in docs, but I found a code which seems to force Trainer to reshuffle using callbacks (so I guess the trainer will not reshuffle the dataset?):github.comhuggingface/community-events/blob/a2d9115007c7e44b4389e005ea5c6163ae5b0470/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py#L549-L557# 11. Configure Trainer# Trainer callback to reinitialise and reshuffle the streamable datasets at the beginning of each epoch# Only required for streaming: Trainer automatically shuffles non-streaming datasetsclass ShuffleCallback(TrainerCallback):def on_epoch_begin(self, args, state, control, train_dataloader, **kwargs):if isinstance(train_dataloader.dataset, IterableDatasetShard):pass  # set_epoch() is handled by the Trainerelif isinstance(train_dataloader.dataset, IterableDataset):train_dataloader.dataset.set_epoch(train_dataloader.dataset._epoch + 1)