Converting an HF dataset to pandas
Wondering if there is a way to convert a dataset downloaded using load_dataset to pandas?
2021-11-19T11:07:38Z
Hi,we have a method for that -Dataset.to_pandas. However, note that this will load the entire dataset into memory by default to create a DataFrame. If your dataset is too big to fit in RAM, load it in chunks as follows:dset = load_dataset(...)
for df in dset.to_pandas(batch_size=..., batched=True):
    # process dataframesAnother option is to use thepandasformatter, which will return a DataFrame object each time the dataset is indexed/sliced:dset = load_dataset(...)
dset.set_format("pandas")
dset[10]  # returns a dataframe with 1 row
dset[10:30]  # returns a dataframe with 20 rows
2024-04-03T08:04:44Z
Just a little add-on: if the Huggingface Dataset consists of train and test data, after the set_format  you should write for example:dataset['train'][10]Otherwise, it returns Invalid key error.
2024-06-17T15:35:38Z
mariosasko:for df in dset.to_pandas(batch_size=..., batched=True):Got the below error when trying the above methodAttributeError: ‘DatasetDict’ object has no attribute ‘to_pandas’