[solved] How to load multiple arrow files into one dataset
HelloI am trying learn how to do embeddings etc. And in an earlier run, had saved some embeddings into arrow files usingdataset.save_to_disk( )which generated two filesdata-00000-of-00002.arrow and data-00001-of-00002.arrowToday, I want to load these two files into one dataset.what Ive’ tried:Dataset.from_file("/content/drive/MyDrive/data-00000-of-00002.arrow", "/content/drive/MyDrive/data-00001-of-00002.arrow"), that is, passing the two files tofrom_filebut this gives me an errorAttributeError: 'str' object has no attribute 'copy'So how do i load these two into one dataset?expected outcome: the orignial dataset that I had created and saved to disk is re-created from these two files.thank youUpdate: Unsure if this is a workaround or something but I usedfrom datasets import load_datasetand thendata = load_dataset("arrow", data_files={file1, file2, file3})instead ofDataset.from_fileand this works. I am kinda glad my question went into the Akismet queue, because it made me want to try alternativesIf someone could tell me how to use theDataset.from_filefunction do the same, I would love that.Cheers
2023-08-25T05:03:35Z
I have the same problem. Load from multiple arrow files into a dataset. Similar to the result after the map(). I want to process the arrow files I need in advance
2023-08-25T08:45:07Z
HI Enze, what do you mean process the files in advance? Are you able to load the files into a dataset? please add some details on what you’re trying to do
2023-08-28T18:22:44Z
You can load  multiple Arrow files usingDataset.from_file(assuming their schemas match, so no additional casts are required) as follows:from datasets import concatenate_datasets, Dataset

ds = concatenate_datasets([Dataset.from_file(arrow_file) for arrow_file in arrow_files])
2023-09-16T19:01:02Z
such as, to processing a dataset into multiple arrow files and then using load_ from_disk methods