I keep getting KeyError:'safe' when loading my datasets
Everything was working well for months and now i am suddenly getting this error everytime i try to load my dataset.  " from datasets import load_datasetdataset = load_dataset(“sebascorreia/sc09”, split=“test”)"/usr/local/lib/python3.10/dist-packages/huggingface_hub/REDACTED_TOKEN.py ininit(self, **kwargs)636         if security is not None:637             security = BlobSecurityInfo(→ 638                 safe=security[“safe”], av_scan=security[“avScan”], pickle_import_scan=security[“pickleImportScan”]639             )640         self.security = securityKeyError: ‘safe’" My dataset is just a replica of the SC09 dataset used by the WaveGAN paper. I dont i changed anything about it, i tried upgrading the datasets and the hugginface hub libraries, I tried loading other datasets and i still get the same error. Am i the one considered unsafe or something? I really need this fixed, i am finishing up my evaluation for my dissertation, this is the last thing i needed right now. If anyone knows what is going on please help!
2024-09-06T17:05:03Z
Im having the same error. Were you able to figure out the issue?
2024-09-06T17:09:53Z
Not yet
2024-09-06T17:24:00Z
Same here. Couldn’t load datasets.
2024-09-06T17:24:26Z
I too am getting the same error when loading my private dataset.
2024-09-06T17:25:33Z
Me too. I’m getting the same error. Trying to load my public dataset.
2024-09-06T17:27:00Z
same ;((((
2024-09-06T17:27:09Z
Got the same issue
2024-09-06T17:27:14Z
Me, too.  Worked yesterday,
2024-09-06T17:28:52Z
Me too, please keep us updated!
2024-09-06T17:32:06Z
Also discussed here:github.com/huggingface/datasetsOlder datasets throwing safety errors with 2.21.0opened04:26PM - 06 Sep 24 UTCalvations### Describe the bug

The dataset loading was throwing some safety errors for…this popular dataset `wmt14`. 

[in]:

```
import datasets

# train_data = datasets.load_dataset("wmt14", "de-en", split="train")
train_data = datasets.load_dataset("wmt14", "de-en", split="train")
val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")
```


[out]:

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-9-445f0ecc4817>](https://localhost:8080/#) in <cell line: 4>()
      2 
      3 # train_data = datasets.load_dataset("wmt14", "de-en", split="train")
----> 4 train_data = datasets.load_dataset("wmt14", "de-en", split="train")
      5 val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")

12 frames
[/usr/local/lib/python3.10/dist-packages/huggingface_hub/REDACTED_TOKEN.py](https://localhost:8080/#) in __init__(self, **kwargs)
    636         if security is not None:
    637             security = BlobSecurityInfo(
--> 638                 safe=security["safe"], av_scan=security["avScan"], pickle_import_scan=security["pickleImportScan"]
    639             )
    640         self.security = security

KeyError: 'safe'
```


### Steps to reproduce the bug

See above.

### Expected behavior

Dataset properly loaded.

### Environment info

version: 2.21.0
2024-09-06T17:36:05Z
I think there is some issue with the blob/data HF is sending over which is leading to this issues.The error comes from: site-packages/huggingface_hub/REDACTED_TOKEN.pyL638         safe=security[“safe”], av_scan=security[“avScan”], pickle_import_scan=security[“pickleImportScan”]security data is a nested dictsecurity{‘hf’: {‘blobId’: ‘d8bdd5db36bc882309fab2fddecea23622de8fe9’, ‘name’: ‘README.md’, ‘safe’: True, ‘indexed’: False, ‘avScan’: {‘virusFound’: False, ‘virusNames’: None}, ‘pickleImportScan’: None}}Due to the nested nature,  security[‘safe’] is not directly accessible. I did a quick fix by setting, security = security[‘hf’], and the code works. This is just a quick fix, but  the right solution should be coming from HF.
2024-09-06T17:44:01Z
Same, i tried many different datasets but same error
2024-09-06T17:44:35Z
Hi,where do you set this ?
2024-09-06T17:44:39Z
Samethis error was not present few hours back. HuggingFace please take a look at it.
2024-09-06T17:45:36Z
hi, would you please instruct how to set it? thanks!
2024-09-06T17:47:19Z
please show how you did it
2024-09-06T17:50:07Z
You should find the huggingface_hub package in site-packages of your virtual env.You can find this path via:import huggingface_hub
print(huggingface_hub.__file__)
venv/name_of_venv/lib/python3.10/site-packages/huggingface_hub/__init__.pyThis command gives you the directory of the package. Under this directory you need to edit REDACTED_TOKEN.py file (Line 637).self.last_commit = last_commit
        security = kwargs.pop("security", None)
        if security is not None:
            #This is the hot-fix
            security=security['hf']
            security = BlobSecurityInfo(
                safe=security["safe"], av_scan=security["avScan"], pickle_import_scan=security["pickleImportScan"]
            )
        self.security = security
2024-09-06T17:58:08Z
Great it works
2024-09-06T18:09:44Z
Hey guys , try:  pip install datasets==2.20.0This will fix the problem
No date available
No reply text available