Using a generator for the map function in an iterable dataset
Hi!I have an audio dataset, and I’m using the standardload_datasetpattern to get the dataset into memory.dataset = load_dataset("audiofolder", data_dir="../data", streaming=streaming)And then I usemapwithbatched=Trueto do preprocessing. The problem is that the preprocessing explodes the size of the dataset due to the batching. Is it possible to pass a generator tomapinstead of a function? That’d fix the problem in that a single sample would be emitted whenevernextis called instead of making the whole batch in one go.I know that it’s possible to create anIterableDatasetfrom a generator, and I can do that as a last resort, but theload_datasetis quite nice and saves me from doing a bunch of manual librosa stuff.Thanks in advance for any tips!
2023-12-30T00:52:46Z
You can reduce thebatch_sizeto emit examples small batches a a time. Or simply not use batching