How to save my tokenizer using save_pretrained?
I have just followedthistutorial on how to train my own tokenizer.Now, from training my tokenizer, I have wrapped it inside a Transformers object, so that I can use it with the transformers library:from transformers import BertTokenizerFast

new_tokenizer = BertTokenizerFast(tokenizer_object=tokenizer)Then, I try to save my tokenizer using this code:tokenizer.save_pretrained('/content/drive/MyDrive/Tokenzier')However, from executing the code above, I get this error:AttributeError: 'tokenizers.Tokenizer' object has no attribute 'save_pretrained'Am I saving the tokenizer wrong?If so, what is the correct approach to save it to my local files, so I can use it later?
2021-08-12T16:54:40Z
You are saving the wrong tokenizer ;-).new_tokenizer.save_pretrained(xxx)should work.
2021-08-12T17:07:18Z
Thank you very much for that! And, one more thing… When I want to use my tokenizer for masked language modelling, do I use the pretrained model notebook?
2021-08-12T17:11:25Z
I’m not sure which notebook you are referencing. If you want to train a language model from scratch on masked language modeling, it’s inthis notebook.
2021-08-12T22:52:06Z
I see - will take a look at that. So, after training my tokenizer, how do I use it for masked language modelling task?
2021-08-13T13:06:43Z
@sguggerDo I replace the following with where I saved my trained tokenizer?model_checkpoint = "bert-base-cased"
tokenizer_checkpoint = "sgugger/bert-like-tokenizer"