Loading SentencePiece tokenizer
When I use SentencePieceTrainer.train(), it returns a .model and .vocab file. However when trying to load it using AutoTokenizer.from_pretrained() it expects a .json file. How would I get a .json file from the .model and .vocab file?
2022-07-26T09:09:45Z
Did you save the model first? You can do that using thesave_pretrained()function, and then simply load the tokenizer by providing the model’s directory (where all the necessary files have been stored) to thefrom_pretrained()function.
2023-10-22T16:51:36Z
@mapama247i am actully stuck here.i tried loading the sentencepiece trained tokenizer using the following scripttok=tokenizers.SentencePieceUnigramTokenizer.from_spm("tokenizer.model")
tok.save_pretrained("REDACTED_TOKEN_tokenizer")I get the following error:AttributeError: 'SentencePieceUnigramTokenizer' object has no attribute 'save_pretrained'
2023-10-24T12:49:07Z
Hi@StephennFernandes,This is because in the previous message I was talking about the AutoTokenizer class from transformers. I cannot reproduce your problem because you’re loading from local, but I see that the SentencePieceUnigramTokenizer that you’re using has a couple methods that you could try:saveandsave_model. You can see them doingdir(tokenizers.SentencePieceUnigramTokenizer)