Add data augmentation process during training every epoch
Hello,I’d like to process my training dataset every epoch.I want to add random processing as data augmentation, and I want to do it during training, not preprocessing.I think I can do it withTrainer,DataCollator, or__getitem__ofdatasets.arrow_dataset.Dataset, but where should I do it?For the evaluation set and test set, I plan to do a preprocess usingdatasets.arrow_dataset.Dataset.map.Thank you in advance.
2021-01-19T21:14:07Z
TheDataCollatorcan help if you have something randomized in the call that returns the batch. Agetitemin your Dataset can also help, it all depends on what you are trying to do exactly.TheTrainerin itself has nothing implemented for data augmentation, so it won’t help you.
2021-01-20T03:52:24Z
Hi@sguggerThank you for the answer!I now understand for my purpose I should treat aDataCollatororgetitemin a Dataset, depending on what I am trying to do, and theTraineris not an appropriate place in which data augmentation is done.Referring to your answer in my another questionHow to use Seq2SeqTrainer (Seq2SeqDataCollator) in v4.2.1 - #5 by sgugger,I’ll consider the best way to implement to get something randomized in my case.Thank you so much!