How to load a huggingface amnesty/qa dataset from local path?
I have downloaded dataset in local directly from github:brew install git-lfs
git lfs instal
git clone https://huggingface.co/datasets/explodinggradients/amnesty_qaIt has following directory structure:image466×456 33.7 KBI want to loadenglish.jsonusing hugging face load_dataset.The json format is like this:{
"question": [..,..,],
"answer": [..,...,],
"contexts": [
        [..], [].. ],
}from datasets import load_dataset
dataset = load_dataset(
    "explodinggradients/amnesty_qa",
    "english_v3",
    trust_remote_code=True
)@vpkprasanna
2024-11-27T04:02:08Z
As long as the path is correct, it seems that no special operation is required, but you may be fine without specifying trust_remote_code. Because it is not a remote code after downloading…huggingface.coLoading methodsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-11-27T14:51:09Z
Not working! Does the codeload_datasetrunsamnesty_qa.pyfile to fetchenglish_v3data?