Getting Value Error while using datasets_interleave_datasets method
Hi, I am following the Hugging Face course. In chapter.5, the Datasets Library, under the topic “Big data? Datasets to the rescue” topic, I am trying to execute the below code in Colab notebook as detailed in the course.combined_dataset = interleave_datasets([pubmed_dataset_streamed, law_dataset_streamed])

list(islice(combined_dataset, 2))In response, I am getting the below error, for which I am unable to find a solution anywhere on the web including Github and Stackoverflow. Appreciate if someone can look into and let me know the resolution. Thank you.---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-65-42004c060c27> in <module>
      2 from datasets import interleave_datasets
      3 
----> 4 combined_dataset = interleave_datasets([pubmed_dataset_streamed, law_dataset_streamed])
      5 list(islice(combined_dataset, 2))

2 frames
/usr/local/lib/python3.8/dist-packages/datasets/features/features.py in _check_if_features_can_be_aligned(features_list)
   2052         for k, v in features.items():
   2053             if not (isinstance(v, Value) and v.dtype == "null") and name2feature[k] != v:
-> 2054                 raise ValueError(
   2055                     f'The features can\'t be aligned because the key {k} of features {features} has unexpected type - {v} (expected either {name2feature[k]} or Value("null").'
   2056                 )

ValueError: The features can't be aligned because the key meta of features {'meta': {'case_ID': Value(dtype='string', id=None), 'case_jurisdiction': Value(dtype='string', id=None), 'date_created': Value(dtype='string', id=None)}, 'text': Value(dtype='string', id=None)} has unexpected type - {'case_ID': Value(dtype='string', id=None), 'case_jurisdiction': Value(dtype='string', id=None), 'date_created': Value(dtype='string', id=None)} (expected either {'pmid': Value(dtype='int64', id=None), 'language': Value(dtype='string', id=None)} or Value("null").
2022-12-21T14:33:32Z
Hi! This error means that the features of the interleaved datasets are not the same, and they have to be for interleaving. To fix it, make surepubmed_dataset_streamed.featuresis equal tolaw_dataset_streamed.featuresbefore theinterleave_datasetscall.
2022-12-21T14:41:19Z
Thanks@mariosaskofor the suggestion. Wondering why this is not mentioned in the course! Nevertheless, how do I equate the features in both the datasets; can you give me the syntax? Thank you.
2022-12-21T15:13:10Z
You can drop themetacolumn (not needed for training) to equate them:combined_dataset = interleave_datasets(
    [
        pubmed_dataset_streamed.remove_columns("meta"),
        law_dataset_streamed.remove_columns("meta")
    ]
)(Please make sure you are using the latest release ofdatasets(2.8.0) before running this; check withimport datasets; print(datasets.__version__))
2022-12-23T08:23:14Z
@mariosako. I am sorry, I am unable to try your suggestion because now I am encountering a value error while loading the dataset using the streaming option itself.
2022-12-23T13:45:16Z
Linking the issue:Getting Value Error while loading a dataset.. · Issue #5388 · huggingface/datasets · GitHub
2023-01-17T06:25:34Z
This does indeed work!However, i am interested to know whylaw_dataset_streamed.featuresdoesn’t return anything.I am instantiating the thelaw_dataset_streamedobject aslaw_dataset_streamed = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)If one cannot see the list of the features of anIterableDatasetthen it’s difficult to know which features to drop.Coincidentally, i can see the features of thepubmed_dataset_streamed.features.datasets.__version__= 2.8Thank you
2023-01-17T12:05:16Z
You can not see the columns of law_dataset_streamed by calling .features method is because law_dataset_streamed is an iterable dataset object (because of streamed=True parameter in the load_dataset() statement) and to explore the elements of an Iterable Dataset we need to iterate over it. We can access the columns of our streamed dataset as follows as the generator yields a dictionary:next(iter(pubmed_dataset_streamed)).keys()dict_keys([‘meta’, ‘text’])
2023-01-17T12:08:26Z
This helps and solves the problem. However, it looks like we shall first install zstandard before importing datasets library for everything to work correctly.
2023-01-17T12:20:59Z
I can indeed see thefeaturesof theIterableDatasetas i can see it forpubmed_dataset_streamed.featuresso i don’t understand why it shouldn’t work forlaw_dataset_streamed
2023-02-06T13:55:13Z
Hi@Shamik! The former dataset has a loading script in which we can define its features in advance, but the latter doesn’t since it reads data from arbitrary local/remote files. Assuming all the rows have the same columns, you can callresolve_features()on it to fix this (otherwise, you can passfeaturestoload_datasetto manually specify the features).
2023-04-08T06:15:19Z
Thanks for your reply.Have created a PR for the sameInterleaving Datasets Bug Fix by Shamik-07 · Pull Request #478 · huggingface/course · GitHub