Load_dataset() doesn't load ONE of the Subset
So I have a dataset:stepkurniawan/qa-rag-llama · Datasets at Hugging Facein that dataset, I made a couple of subsets:“default”,“Llama-2-13b-chat-hf”, and“Llama-2-7b-chat-hf”.I load using :qa_dataset = load_dataset("stepkurniawan/qa-rag-llama","Llama-2-13b-chat-hf")When I load “Llama-2-7b-chat-hf”, everything works well, however, when I try to load “Llama-2-13b-chat-hf”, it gave me the dataset of the “default” one.Why?And how can I fix it?Thank you!Update: now I changed the “Llama-2-7b-chat-hf”, the changes are seen in the Dataset Viewer, but when I do in the codeHF_HUB_QA_LLAMA = "stepkurniawan/qa-rag-llama"
qa_dataset = load_dataset(HF_HUB_QA_LLAMA, "Llama-2-7b-chat-hf")it still get me the old dataset… I dont understand…Update 2:The difference is just the number of rows… Previously, they have only 3-5 rows… But after I change, everything becomes 50 rows. Does HF have rows limit ?
2023-10-18T13:36:24Z
pushing for visibility
2023-10-20T09:15:03Z
Anyone?is this a bug?
2023-10-20T10:10:54Z
Hi ! Maybe you tried to runpush_to_hubtwice at the same time and therefore it caused conflicts ? Feel free to retry and push subsets one at a time.Does HF have rows limit ?No, you can have as many rows as you want
2023-10-20T10:18:41Z
Hi@lhoestq, thanks for answeringIm sure I didnt do double pushing at the same time.I tried with 2 subsets (“Llama2-13b” & “Llama2-7b”) and both of them are behaving weird the same way.can you confirm what I am doing is correct?push_to_hub:result_dataset.push_to_hub(HF_HUB_QA_LLAMA, token=REDACTED_TOKEN, config_name=subset_name)load_dataset :qa_dataset = load_dataset(HF_HUB_QA_LLAMA, subset_name)And another question is, by best practice, do you recommend push_to_hub(), or directly uploading to HF using terminal?
2023-10-20T10:31:36Z
Ok cool ! What you’re doing is correct.And another question is, by best practice, do you recommend push_to_hub(), or directly uploading to HF using terminal?push_to_hub()is the recommended way. It optimizes for I/O efficiency, allows resuming, offers a better experience for the dataset viewer, and also allows defining multiple subsets.
2023-10-20T10:33:07Z
Can you try to reupload your subsets and ping me if you still have an issue when you reload the data usingload_dataset?
2023-10-20T10:33:38Z
ok…that is a bad news…since now I dont know what I did wrong and its still unsolvedIt normally works with datasets without subset, however im struggling to do with with subset…I can try do it once more and ping you if the bad news still persist, sure.
2023-10-20T16:34:23Z
@lhoestqHey, Ive tested it again.It turns out that using another laptop will give me 50 rows, and using my laptop, I still get 3 rows.Does it mean I have to clear cache somehow/ somewhere?
2023-10-20T17:05:36Z
The cache is automatically refreshed as long as you have an internet connection: it will get the latest files from the HF website. Otherwise it can use the lcoal cache and show a WARNING log message to say that it loaded the dataset from your local cache
2023-10-20T17:49:20Z
I certainly don’t get the warning.qa_dataset = load_dataset("stepkurniawan/qa-rag-llama", "Llama-2-13b-chat-hf")
print("something")and this is in my terminal:(base) [xxx@ml3-gpu2 RAG-comparation]$ /usr/bin/env /home/xxx/python /home/xxx/RAG-comparation/rag_ragas.py 
somethingbtw, how to do I delete the cache?
2023-10-24T08:58:44Z
Actually it’s at “info” log level
2023-10-24T18:29:23Z
@lhoestqI turned on INFO log level usingtransformers.logging.set_verbosity_info()But i still didnt get the cache warning. Is the code above correct?It could be that the problem is not the cache.We could check that by deleting the cache maybe?How to do that?
2023-10-25T10:12:19Z
The cache is at~/.cache/huggingface/datasetsby default, feel free to delete it (make sure you’re not usingdatasetswhen deleting it)
2023-10-25T10:20:07Z
Thank you! it works after I delete the cache!!!But it is weird since I dont want my code to use cache…Is there any way i can force the program to NOT use the cache?
2023-10-25T10:41:00Z
yes you can doimport datasets

datasets.disable_caching()andload_dataset(..., download_mode="force_redownload")