How to resolve the hugging face error ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils'?
I was trying to use the ViTT transfomer. I got the following error with code:from pathlib import Path
import torchvision
from typing import Callable
root = Path("~/data/").expanduser()
# root = Path(".").expanduser()
train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)
img2tensor: Callable = torchvision.transforms.ToTensor()
from transformers import ViTFeatureExtractor
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
x, y = train_ds[0]
print(f'{y=}')
print(f'{type(x)=}')
x = img2tensor(x)
x = x.unsqueeze(0)  # add batch size 1
out_cls: ImageClassifierOutput = model(x)
print(f'{out_cls.logits=}')errorFiles already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 11, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/__init__.py", line 30, in <module>
    from . import dependency_versions_check
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 36, in <module>
    from .utils import is_tokenizers_available
ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils' (/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/utils/__init__.py)I tried upgrading everything but it still failed. Upgrade commands:/Users/brandomiranda/opt/anaconda3/envs/meta_learning/bin/python -m 
pip install --upgrade pip
pip install --upgrade pip
pip install --upgrade huggingface-hub

pip install --upgrade transformers
pip install --upgrade huggingface-hub
pip install --upgrade datasets

pip install --upgrade tokenizers

pip install pytorch-transformers

pip install --upgrade torch
pip install --upgrade torchvision
pip install --upgrade torchtext
pip install --upgrade torchaudio

# pip install --upgrade torchmeta
pip uninstall torchmetaWhy and how to fix it?Pip list:(meta_learning) ❯ pip list
Package                                           Version    Editable project location
------------------------------------------------- ---------- ------------------------------------------------------------------------------
absl-py                                           1.0.0
aiohttp                                           3.8.1
aiosignal                                         1.2.0
antlr4-python3-runtime                            4.8
argcomplete                                       2.0.0
async-timeout                                     4.0.1
attrs                                             21.4.0
automl-meta-learning                              0.1.0      /Users/brandomiranda/automl-meta-learning/automl-proj-src
bcj-cffi                                          0.5.1
boto                                              2.49.0
boto3                                             1.24.85
botocore                                          1.27.85
Bottleneck                                        1.3.4
Brotli                                            1.0.9
brotlicffi                                        1.0.9.2
brotlipy                                          0.7.0
cachetools                                        4.2.4
certifi                                           2022.9.14
cffi                                              1.15.1
charset-normalizer                                2.0.9
cherry-rl                                         0.1.4
click                                             8.0.3
cloudpickle                                       2.0.0
colorama                                          0.4.4
configparser                                      5.2.0
conllu                                            4.4.1
crcmod                                            1.7
cryptography                                      37.0.1
cycler                                            0.11.0
Cython                                            0.29.25
dataclasses                                       0.6
datasets                                          2.5.1
dill                                              0.3.4
diversity-for-predictive-success-of-meta-learning 0.0.1      /Users/brandomiranda/diversity-for-predictive-success-of-meta-learning/div_src
docker-pycreds                                    0.4.0
editdistance                                      0.6.0
et-xmlfile                                        1.1.0
fairseq                                           0.10.0
fastcluster                                       1.2.4
fasteners                                         0.17.3
filelock                                          3.6.0
fonttools                                         4.28.3
frozenlist                                        1.2.0
fsspec                                            2022.7.1
gcs-oauth2-boto-plugin                            3.0
gitdb                                             4.0.9
GitPython                                         3.1.24
google-apitools                                   0.5.32
google-auth                                       2.3.3
google-auth-oauthlib                              0.4.6
google-reauth                                     0.1.1
grpcio                                            1.42.0
gsutil                                            5.6
gym                                               0.21.0
h5py                                              3.6.0
higher                                            0.2.1
httplib2                                          0.20.4
huggingface-hub                                   0.10.0
hydra-core                                        1.1.1
idna                                              3.3
importlib-metadata                                4.11.3
jmespath                                          1.0.1
joblib                                            1.1.0
kiwisolver                                        1.3.2
lark-parser                                       0.12.0
learn2learn                                       0.1.7
lxml                                              4.8.0
Markdown                                          3.3.6
matplotlib                                        3.5.1
mkl-fft                                           1.3.1
mkl-random                                        1.2.2
mkl-service                                       2.4.0
monotonic                                         1.6
multidict                                         5.2.0
multiprocess                                      0.70.12.2
multivolumefile                                   0.2.3
munkres                                           1.1.4
networkx                                          2.6.3
numexpr                                           2.8.1
numpy                                             1.21.5
oauth2client                                      4.1.3
oauthlib                                          3.1.1
omegaconf                                         2.1.1
openpyxl                                          3.0.10
ordered-set                                       4.0.2
packaging                                         21.3
pandas                                            1.4.2
pathtools                                         0.1.2
Pillow                                            9.0.1
pip                                               22.2.2
plotly                                            5.4.0
portalocker                                       2.3.2
progressbar2                                      3.55.0
promise                                           2.3
protobuf                                          3.19.1
psutil                                            5.8.0
py7zr                                             0.16.1
pyarrow                                           9.0.0
pyasn1                                            0.4.8
pyasn1-modules                                    0.2.8
pycparser                                         2.21
pycryptodomex                                     3.15.0
pyOpenSSL                                         22.0.0
pyparsing                                         3.0.6
pyppmd                                            0.16.1
PySocks                                           1.7.1
python-dateutil                                   2.8.2
python-utils                                      2.5.6
pytorch-transformers                              1.2.0
pytz                                              2021.3
pyu2f                                             0.1.5
PyYAML                                            6.0
pyzstd                                            0.14.4
qpth                                              0.0.15
regex                                             2021.11.10
requests                                          2.28.1
requests-oauthlib                                 1.3.0
responses                                         0.18.0
retry-decorator                                   1.1.1
rsa                                               4.7.2
s3transfer                                        0.6.0
sacrebleu                                         2.0.0
sacremoses                                        0.0.46
scikit-learn                                      1.0.1
scipy                                             1.7.3
seaborn                                           0.11.2
sentencepiece                                     0.1.97
sentry-sdk                                        1.5.1
setproctitle                                      1.2.2
setuptools                                        58.0.4
shortuuid                                         1.0.8
six                                               1.16.0
sklearn                                           0.0
smmap                                             5.0.0
subprocess32                                      3.5.4
tabulate                                          0.8.9
tenacity                                          8.0.1
tensorboard                                       2.7.0
tensorboard-data-server                           0.6.1
tensorboard-plugin-wit                            1.8.0
termcolor                                         1.1.0
texttable                                         1.6.4
threadpoolctl                                     3.0.0
tokenizers                                        0.13.0
torch                                             1.12.1
torchaudio                                        0.12.1
torchtext                                         0.13.1
torchvision                                       0.13.1
tornado                                           6.1
tqdm                                              4.62.3
transformers                                      4.22.2
typing_extensions                                 4.3.0
ultimate-anatome                                  0.1.1      /Users/brandomiranda/ultimate-anatome
ultimate-aws-cv-task2vec                          0.0.1      /Users/brandomiranda/ultimate-aws-cv-task2vec
ultimate-utils                                    0.6.1      /Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src
urllib3                                           1.26.11
wandb                                             0.13.3
Werkzeug                                          2.0.2
wheel                                             0.37.0
xxhash                                            2.0.2
yarl                                              1.8.1
yaspin                                            2.1.0
zipp                                              3.8.0hugging face (HF) related gitissues:Import Error : cannot import name 'create_repo' from 'huggingface_hub' · Issue #15062 · huggingface/transformers · GitHubTokenizer import error · Issue #120 · huggingface/tokenizers · GitHubThe Conda package doesn't work on CentOS 7 and Ubuntu 18.04 · Issue #585 · huggingface/tokenizers · GitHubFailed to import transformers · Issue #11262 · huggingface/transformers · GitHubSO related:python - Transformer: Error importing packages. "ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler'" - Stack OverflowSO:python - How to resolve the hugging face error ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils'? - Stack Overflow
2022-10-06T07:55:47Z
Have you tried running the code without Eclipse/Pycharm ? It seemspydevuses some kind of magic in betweentransformerscalls. Have you also tried uninstalling (making sure the version was uninstalled) and then reinstalling transformers ?Nb: This is probably not linked to CentOS or any of the related issues. Also please try not to spam everywhere at once with your questions, you are not maximizing your chances of getting a good answer.
2023-04-10T12:52:57Z
I had the same issue and fixed it by installing an older version of the transformers library as i got this error:ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.spacy-transformers 1.2.2 requires transformers<4.27.0,>=3.4.0, but you have transformers 4.27.0 which is incompatible.It auto installed transformers==4.27.4
2023-08-16T09:30:19Z
had same issue, after downgrading the transformers version to 4.27.4, Its working fine. Thanks@alexthevries
2023-09-28T18:36:57Z
Which version did you downgrade to?
2024-10-21T00:30:02Z
Hello @I’m having the same issues. I can see people proposing to downgrade the transformers but some packages need an update transformers. Do you have any others options?Thanks
2024-10-21T08:55:26Z
The most positive way to deal with this is to update the libraries that require the old transformers library, but there are some libraries that have not been updated for several years…For libraries that many people have trouble without, there may be patches available by other developers on github or pip to fix them.The last resort is to rewrite the library itself.