How to load dataset that exist in cache path
Hi,I try this code in a server with internet connection:from datasets import load_dataset
wiki = load_dataset("wikipedia", "20200501.en", split="train")Then automatic downloading process began and there is a folder~/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63/which containswikipedia-train.arrowand some other files.Now I’d like to use the dataset in a server without internet connection.What should I do?I tried it withfrom datasets import load_from_disk
 wiki = load_from_disk("~/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63"It showedstate.jsonnot found in that folder.Any advice?
2021-02-18T15:27:13Z
solved it bywiki = load_dataset("~/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63/wikipedia.py", "20200501.en", split="train")
2021-02-19T15:04:05Z
Hi !Glad you managed to solve your issueIn the recently released 1.3.0 version of datasets, you should also be able to reload your dataset with no internet connection withwiki = load_dataset("wikipedia", "20200501.en", split="train")Indeed if there’s no internet connection, this will fall back on the latest wikipedia dataset that you’ve loaded withload_dataset, and notify you with a warning.In your case it would look for a processed dataset in “~/.cache/huggingface/datasets/wikipedia/20200501.en”Let me know if you have other questions !
2021-02-24T06:56:45Z
Hi,I upgrade to 1.3.0.And it indeed works. Thanks!
2023-12-06T16:25:20Z
Hi,I was wondering what happens when the dataset is already cached but the computer has access to the internet and we use load_dataset() as you did. Will it use from the cache or reload again?Thank you
2023-12-06T16:44:57Z
It will reload from cache(not super important but there is one bug that will be fixed in the next release:Datasets created with `push_to_hub` can't be accessed in offline mode · Issue #3547 · huggingface/datasets · GitHub)