How to merge two dataset objects?
Hi everyone!I have two datasets, loaded as CSV files, which have the same features/columns. I would like to know if there is a way to merge both datasets into a larger one (like I would do withpd.concat((df_1, df_2))using pandas.In case that such method does not exist, would it be interesting to implement such functionality?Thanks in advance
2020-08-24T19:18:55Z
I would rather combine the csv’s
2020-08-24T19:30:42Z
Are you usingnlp?If so, you could trynlp.concatenate_datasets
2020-08-24T21:06:34Z
Thank you. That is exactly what I was looking for, but I couldn’t find it in the documentation (now that I now the method I can find it in the API when I autocomplete the code, but it doesn’t appear anywhere in the documentation).
2021-05-03T17:53:03Z
Quick update since I see that this thread still has views:concatenate_datasetsis available through thedatasetslibraryhere, since the library was renamed.
2021-06-25T08:58:21Z
Hi@lhoestq, thanks for the solution. I follow that approach but getting errors to merge two datasetsdataset_ar = load_dataset('wikipedia',language='ar', date='20210320', beam_runner='DirectRunner')
dataset_bn = load_dataset('wikipedia',language='bn', date='20210320', beam_runner='DirectRunner')I tried two ways to concatenate but both approaches give errors. Could you please help to find out what am I missing? ThanksFirst Approachdataset_cc = concatenate_datasets(dataset_ar, dataset_bn)Traceback (most recent call last):File “”, line 1, inFile “/home/anaconda3/envs/nlp/lib/python3.9/site-packages/datasets/arrow_dataset.py”, line 3135, in concatenate_datasetsif axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):File “/home/anaconda3/envs/nlp/lib/python3.9/site-packages/datasets/arrow_dataset.py”, line 3135, inif axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):AttributeError: ‘str’ object has no attribute ‘features’Second Approachdataset_cc = concatenate_datasets(dataset_ar['train'], dataset_bn['train'])Traceback (most recent call last):File “”, line 1, inFile “/home/anaconda3/envs/nlp/lib/python3.9/site-packages/datasets/arrow_dataset.py”, line 3135, in concatenate_datasetsif axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):File “/home/anaconda3/envs/nlp/lib/python3.9/site-packages/datasets/arrow_dataset.py”, line 3135, inif axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):AttributeError: ‘dict’ object has no attribute ‘features’
2021-06-25T09:03:49Z
I solve this issue, it was a silly mistakeSolution:dataset_cc = concatenate_datasets([dataset_ar['train'], dataset_bn['train']])
2024-02-28T10:04:00Z
concatenate_datasets([part1, part2, part3]) this way~