How to filter Datasets object?
I have a hugging face dataset object and I want to filter it as I would a pandas dataframe:train[train['language']=="English"]‘language’ is one of the features in the train split.I also tried:train.select(train['language']=='English')and got error:TypeError: ‘bool’ object is not iterableAnd similarly, boolean masking, with the same error code:is_english = dataset['train']['language'] == 'English'
eng_convos = dataset['train'].select(is_english)Or a list comprehnsion:eng_convos = [dataset['train']['conversation'] for dataset['train']['conversation'] in dataset['train'] if dataset['train']['language']=='English']I got:TypeError: ‘Dataset’ object does not support item assignmentLastly, I tried follwoing advice from this discussion postFiltering Datasetimport pyarrow as pa

import pyarrow.compute as compute

table = dataset.data

flags = compute.is_in(train['language'], value_set=pa.array(['English'], pa.string()))

filtered_table = train.filter(flags)

filtered_table.to_pandas()and got error:     557 # apply actual function→ 558 out: Union[“Dataset”, “DatasetDict”] = func(self, *args, **kwargs)559 datasets: List[“Dataset”] = list(out.values()) if isinstance(out, dict) else [out]560 # re-apply format to the outputFile c:\Users\Admin\HC3 EDA.venv\Lib\site-packages\datasets\fingerprint.py:482, in fingerprint_transform.._fingerprint..wrapper(*args, **kwargs)478             validate_fingerprint(kwargs[fingerprint_name])480 # Call actual function→ 482 out = func(dataset, *args, **kwargs)484 # Update fingerprint of in-place transforms + update in-place history of transforms…6260 else:6261     # inputs is a list of columns6262     columns: List[List] = inputsTypeError: ‘pyarrow.lib.BooleanArray’ object is not callable
2024-06-06T15:52:30Z
All I want to do is essentially filter my HF Datasets object as I would a pandas dataframe i.e. only return the dataset where a specific column value meets my condition…
2024-06-06T17:04:12Z
Found a solution:converted the datasets object to pandas DataFrameimport pandas as pd
df = pd.DataFrame(dataset['train'])filtered it as usualenglish_only = df[df['language']=="English"]
2024-06-07T05:04:31Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.