I can't concatenate_datasets because features are not sorted. How do I sort it?
Hi guys,I’m trying to concatenate two datasets that share some common features.But these two datasets have features in a different order.It’s like:DatasetDict({
    train: Dataset({
        features: ['__index_level_0__', 'answers', 'context', 'document_id', 'id', 'question', 'title'],
        num_rows: 3952
    })
    validation: Dataset({
        features: ['__index_level_0__', 'answers', 'context', 'document_id', 'id', 'question', 'title'],
        num_rows: 240
    })
})andDatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 60407
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 5774
    })
})so I erased uncommon features like:__index_level_0__,document_id, etc by using.remove_columnsNow the two DatasetDicts have the same features. However the order is different.1st DatasetDict:features: ['answers', 'context', 'question', 'title'],2nd DatasetDict:features: ['title', 'context', 'question', 'answers'],So when I try to concatenate them by usingdatasets.concatenate_datasets([1stDatasetDict, 2ndDatasetDict]),I get an error that says:ValueError: Features must match for all datasets
2021-05-19T12:36:06Z
hey@jeffnlp, i don’t think you can concatenateDatasetDictobjects withconcatenate_datasets- as described in thedocsthis function expects a list ofDatasetobjects.what happens if you try iterating over bothDatasetDictobjects and building up a new one that concatenates theDatasetsobjects as follows:ds1 = DatasetDict(...)
ds2 = DatasetDict(...)
# Create empty DatasetDict
ds3 = DatasetDict()

for (split1, x), (split2, y) in zip(ds1.items(), ds2.items()):
    ds3[split1] = concatenate_datasets([x, y])this should work sinceconcatenate_datasetscan handle out-of-order columns. if not, you might have a problem with theanswerscolumns if they’re nested and the sub-columns don’t match. in that case you might be better off flattening the columns of each dataset and casting the features into the same types (see e.g.herefor casting details)
2021-07-25T04:12:43Z
Order does matter apparently ! Find the arrow_dataset.py file (it can be found in Lib\site-packages\datasets-1.8.1.dev0-py3.9.egg\datasets); look up line 3181 of the file (my version of the package istransformers==4.9.0dev0) :if axis == 0 and not all([dset.features.type == dsets[0].features.type for dset in dsets]):
        raise ValueError("Features must match for all datasets")
elif axis == 1 and not all([dset.num_rows == dsets[0].num_rows for dset in dsets]):
        raise ValueError("Number of rows must match for all datasets")dset.features.typeis sensitive to the order of the columns; it gives a StructType, as below, which is clearly order-sensitive:pawsqqp['train'].column_names
output : ['idx', 'question1', 'question2', 'label']
pawsqqp['train'].features.type
output: StructType(struct<idx: int32, question1: string, question2: string, label: int64>)So I simply changed that first line in the arrow_dataset.py file toif axis == 0 and not all([dset.features == dsets[0].features for dset in dsets]):and now it no longer cares about the order of the columns, and I can confirm that it’s working well in the examples I had. But I don’t recommend applying this change without a double-check! Make sure this modification works well for your datasets too.
2021-08-11T06:22:13Z
Given two datasets:d1 and d2having the same features but differently orderedStructType, a slightly non-intrusive way might be to cast the data features into the same type (as mentioned by@lewtun)d1 = d1.cast(d2.features)
d3 = datasets.concatenate_datasets([d1, d2])