Pandas dataframe save to dataset gives error
I was looking for a way to write a logfile after my model has made a prediction but got stuck as the append mode is not supported by the file systemI tried to make a dataframe and save it - it is able to read and load the dataframe but unable to save itI am following :Interact with the Hub through the Filesystem APIfrom huggingface_hub import HfFileSystemimport pandas as pdtoken=‘REDACTED_TOKEN’fs = HfFileSystem(token=token)df = pd.read_csv(“hf://datasets/sujitb/data/querylog.csv”)till this it works finedf.to_csv(“hf://datasets/sujitb/data/querylog.csv”)throws errorRepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-66098a8a-2f8d371169e6f6991a126d89;3366d0ed-0e17-4061-8f41-845dfdc872a4)Repository Not Found for url:https://huggingface.co/api/datasets/sujitb/data/preupload/main.Please make sure you specified the correctrepo_idandrepo_type.If you are trying to access a private or gated repo, make sure you are authenticated.Invalid username or password.Note: Creating a commit assumes that the repo already exists on the Huggingface Hub. Please usecreate_repoif it’s not the case.Has anyone faced this or got around this issue?
2024-04-03T12:24:36Z
Hi ! you need to login usinghuggingface-cli logincommand or pass your token asstorage_options(it’s just a dict containing your token)token=‘REDACTED_TOKEN’
fs = HfFileSystem(token=token)
df.to_csv(“hf://datasets/sujitb/data/querylog.csv”, storage_options=fs.storage_options)Otherwisedf.to_csvwill use a brand newHfFileSystem()without token to upload the file and fail
2024-04-03T17:39:02Z
Thanks a lot for your response !  Passing the storage_options solved my problem. I just passed a dict {“token”:‘xxxxxxxxxxx’}  and it worked.
2024-04-04T05:39:55Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.