Preprocessing & augmentation during training
Hi!I am currently fine-tuning a Wav2Vec2 model and I would like to know how I could modify training samples.In Tensorflow it’s possible to do this by callingdataset.map()but doing this with adatasets.arrow_dataset.Datasetwill write a cache file. Besides that, it’s static and unnecessary.So, this here is not an option:train_dataset = train_dataset.map(preprocess_and_augment)There is thedata_collatorand one could do it here. However, this is very likely not very efficient and I would have to take care of parallelization myself.I’m looking for a similar way to process input examples likeDataset.map().
2022-02-09T16:14:04Z
Hi! I thinkDataset.set_transformis what you are looking for.
2022-02-10T08:08:37Z
Yes, this would be something I’d like to do. The problem is that I am having adatasets.arrow_dataset.Dataset$ print(train_dataset)
<class 'dataset.arrow_dataset.Dataset'>which is what I get after callingbuilder_instance.as_dataset(..)following this description:Writing a dataset loading script — datasets 1.11.0 documentationI have tried to call:datasets.Dataset(train_dataset)`giving meTypeError: Expected a pyarrow.Table or a `datasets.table.Table` objectanddatasets.table.Table(train_dataset)throwsAttributeError: 'Dataset' has no attribute 'schema'I guess my question is: How can I get adatasets.Datasetfrom adataset.arrow_dataset.Dataset?
2022-02-10T08:29:22Z
I figured that I can do this:train_dataset = datasets.Dataset(train_dataset.data)
train_dataset.set_transform(example_adapter.preprocess)It’s a bit weird and I am not sure how efficient this is e.g. if I’d have to do more expensive computations though.