From Pandas Dataframe to Huggingface Dataset
Hello everyone,I am doing a tutorial on how to finetune pretrained Sentiment Analysis Classifier and all the finetuning part is based on a HuggingFace Dataset. Is there a way to transform a pandas Dataframe to a HuggingFace Dataset? Would help me alot with my data preprocessing…
2021-08-18T10:35:56Z
You can have a look at here:link
2021-08-18T12:29:19Z
Thanks for your help! Now it works
2022-02-23T06:57:40Z
is there a way to load this into thetrainsplit and another dataframe in memory into thevalidationsplitNone of the following options seem to do the trick:dataset = Dataset.from_pandas(df)
dataset = Dataset.from_pandas(df, split='train')
dataset = Dataset.from_pandas(df, split=NamedSplit('train'))
dataset = Dataset.from_pandas(df, split=datasets.Split.TRAIN)
print(dataset)The best I could come up that worked was (not sure if there is a easier/right way):import pandas as pd
import datasets
from datasets import Dataset, DatasetDict


tdf = pd.DataFrame({"a": [1, 2, 3], "b": ['hello', 'ola', 'thammi']})
vdf = pd.DataFrame({"a": [4, 5, 6], "b": ['four', 'five', 'six']})
tds = Dataset.from_pandas(tdf)
vds = Dataset.from_pandas(vdf)


ds = DatasetDict()

ds['train'] = tds
ds['validation'] = vds

print(ds)
2022-02-23T13:11:08Z
Hi@akomma! Yes, your second approach is the correct one.
2023-12-02T09:51:30Z
Page is not there. Could you please add a page link to that again or page name ?
2023-12-03T09:01:17Z
Look for the from_pandas method atlink
2024-06-18T14:39:02Z
thislinkshould work
2024-07-03T16:21:58Z
Thank you it is helpful
2024-12-20T15:49:57Z
Hi@mariosasko! What did you mean by the second approach?dataset = Dataset.from_pandas(df, split='train')still doesn’t work in 2024. Do I still have to create it with something like this:dataset = DatasetDict({"train": tds, "val": vds})?