How to add new tokens for existing masked language modelling?
Hi,I have followed this tutorial from GitHub on masked language modelling:notebooks/language_modeling.ipynb at master · huggingface/notebooks · GitHubBut, I am wondering, how do I modfiy this code below for the masked language modelling task, and where in my code do I place it?In the tutorial, this line of code is used:from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)This is the code I need to modify to satisfy MLM:Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.
2021-06-11T10:42:05Z
First of all, I guess you want to useBertForMaskedLMinstead ofBertModel. The other parts should work AFAIK.
2021-06-11T11:55:06Z
@BramVanroyYes, I want to useBertForMaskedLM
2021-06-11T13:44:45Z
I have modified the code, but I am getting this error:NameError: name 'BertTokenizer' is not defined