How to load a huggingface dataset from local path?
Take a simple example in this website,https://huggingface.co/datasets/Dahoas/rm-static:if I want to load this dataset online, I just directly use,from datasets import load_dataset
dataset = load_dataset("Dahoas/rm-static")What if I want to load dataset from local path, so I download the files and keep the same folder structure from webFiles and versionsfristly,-data
|-test-00000-of-00001-bf4c733542e35fcb.parquet
|-train-00000-of-00001-2a1df75c6bce91ab.parquet
-.gitattributes
-README.md
-dataset_infos.jsonThen,put them into my folder, but shows error when loading,dataset_path ="/data/coco/dataset/Dahoas/rm-static"
tmp_dataset = load_dataset(dataset_path)It shows,FileNotFoundError: No (supported) data files or dataset script found in /data/coco/dataset/Dahoas/rm-static.
2023-09-01T06:14:42Z
if you want to load dataset from your local path you should follow the below apporachsee the docswhich will accept a parameter named path where apyto process your local file path
2023-09-01T06:24:11Z
thanks for your reply, but, for example, there is no .py file in Dahoas/rm-static, and even though i put README.md, gitattributes in “/data/coco/dataset/Dahoas/rm-static”, it sitll shows aboved error
2023-09-01T06:51:08Z
I solved this problem,data_files = {“train”:“train-00000-of-00001-2a1df75c6bce91ab.parquet”,“test”:“test-00000-of-00001-8c7c51afc6d45980.parquet”}raw_datasets = load_dataset(“parquet”, data_dir=‘/Your/Path/Dahoas/rm-static/data’, data_files=data_files)
2024-06-02T06:47:33Z
Thank you!!!
2024-07-18T07:09:12Z
Good solution! Bro