Where are the actual files to download?
Maybe I don’t understand the structure you have on HuggingFace, but when I download all the files fromcerebras/SlimPajama-627B at mainI get merely 8GB of files. Where are the 895GB that are supposed to be in the dataset? Was I downloading just a preview?
2024-01-08T08:05:16Z
All HF libraries use a shared cache. Read all about it atManage huggingface_hub cache-systemandCache managementfor the Datasets library specifically.
2024-01-08T19:42:03Z
Thank you,@nielsr, but I need to download the dataset just once, so I don’t need to use cache.I even don’t want to store the dataset locally on disk, but to stream it directly from HF to my DropBox account, using Java.All I need is to know is where all the data set files are. The files at the SlimPajama page are only a small preview, so I wonder where the actual dataset is.
2024-01-08T20:14:00Z
hi@DR1912,You could set a different cache dir during dataset loadfrom datasets import load_dataset
dataset = load_dataset('LOADING_SCRIPT', cache_dir="PATH/TO/MY/CACHE/DIR")ref1:https://huggingface.co/docs/datasets/v2.15.0/en/cache#cache-directoryref2:https://huggingface.co/docs/datasets/v2.15.0/en/package_reference/loading_methods#datasets.load_dataset.cache_dirref3:https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhome
2024-01-08T20:29:27Z
Thank you,@radames, but I don’t want to store the dataset in ANY cache folder. I want to stream it in memory from HF directly to DropBox website. I know how to do it, I just don’t know where the files on HF are.
2024-01-08T20:35:13Z
hi@DR1912, I think all files are split across folders, you can resolve then and stream it individuallyhttps://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/main/eg:https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/main/test/chunk1/example_holdout_0.jsonl.zst
2024-01-08T20:47:48Z
Hey@radames, I have already done exactly that, but the files there sum up to only about 8GB, as opposed to the size of about 900GB that they are supposed to be, so I figured that these are just a small preview, and the actual dataset is somewhere else.
2024-01-08T21:20:18Z
if load it viastream=True, this is the metadata, this is all data on that repo. have you tried to decompress the.zstfiles to check the total size?IterableDatasetDict({
    train: IterableDataset({
        features: Unknown,
        n_shards: 118332
    })
    validation: IterableDataset({
        features: Unknown,
        n_shards: 31428
    })
    test: IterableDataset({
        features: Unknown,
        n_shards: 31411
    })
})Please open a discussion herecerebras/SlimPajama-627B · Discussionsand tag the author@rskuzmaAnd different from the original dataset, RedPajama, it contains a dataset script to download the data, but SlimPajama doens’t have it, so all data is on that dataset repo.ref:RedPajama-Data-V2.py · togethercomputer/RedPajama-Data-V2 at main