AutoTokenizer keeps redownloading
Everytime I instantiate the tokenizer usingself.tokenizer = AutoTokenizer.from_pretrained("'sentence-transformers/all-MiniLM-L12-v2'")it calls to huggingface to download the tokenizer:[connectionpool.py:474] -https://huggingface.co:443"HEAD /sentence-transformers/all-MiniLM-L12-v2/resolve/mainIt seems it is not being cached locally and you cannot specify acache_dirfor the tokenizer. You can withAutoModel, but not withAutoTokenizerHow do I prevent this?
2024-03-13T14:48:37Z
You could use tokenizer.save_pretrained() for the same