How to load cached dataset offline?
Hello, all!My computer doesn’t have internet connection. So I have to first download dataset on another computer and copy the dataset to my offline computer.I use the following code snippet to download wikitext-2-raw-v1 dataset.from datasets import load_dataset
datasets = load_dataset("wikitext", "wikitext-2-raw-v1")And I found that some cached files are in the~/.cache/huggingface/'s sub dirs.In the~/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126dir I can see:__init__.py,__pycache__,dataset_infos.json,wikitext.json,wikitext.pyIn the~/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a1 26dir I can see:LICENSE  dataset_info.json  wikitext-test.arrow  wikitext-train.arrow  wikitext-validation.arrowDo I have to copy all those files to the offline computer? Can I changea241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126to other names?Or how to change thosearrowfiles tocsvfiles?
2022-05-27T12:02:59Z
Hi! You only need thearrowfiles, but instead of looking for them in cache, it’s more convenient to save the dataset to disk withsave_to_diskand transfer the generated folder to another computer, where you can simply load the dataset withload_from_disk("path/to/folder").Or how to change thosearrowfiles tocsvfiles?You can useDataset’sto_csvmethod for that.
2022-05-29T13:07:57Z
Thanks for helping me!