How to test masked language model after training it?
Hi,I have followed and trained my masked language model using this tutorial:notebooks/language_modeling.ipynb at master · huggingface/notebooks · GitHubNow, once the model as been saved using this code below:trainer.save_model("my_model")But, the notebook does not seem to include any code to allow me to test my model, so I am unsure how to do this.I have saved my model, but I now want to mask a sentence using my model by doing something like this:The [MASK] of France is ParisThanks!
2021-06-21T13:14:07Z
You can load it in a pipeline by using the folder where you saved it:mask_filler = pipeline("fill-mask", model="my_model")
2021-06-21T13:15:37Z
Thanks. What about the tokenizer? Does that need to be somewhere?
2021-06-21T13:16:59Z
It should be saved in the same folder, which will be the case if you passed it to thetrainer.
2021-06-21T13:28:31Z
Okay, I tried that code you suggested, but I get his error:OSError: Can't load tokenizer for '/content/my_model'. Make sure that:

- '/content/my_model' is a correct model identifier listed on 'https://huggingface.co/models'

- or '/content/my_model' is the correct path to a directory containing relevant tokenizer files
2021-06-21T13:31:15Z
That means you did not save your tokenizer in that folder (which also means you did not pass it when creating yourTrainer).
2021-06-21T13:36:46Z
Oh… Well, this is my trainer:trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["test"],
    data_collator=data_collator,
)I thought thedata_collatorcontains the tokenizer?Like, here, for example:from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
2021-06-21T13:55:53Z
hey@anon58275033it’s true that the data collator uses a tokenizer to perform the collation, but you need to provide thetokenizerargument explicitly to the trainer:trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["test"],
    data_collator=data_collator,
    tokenizer=tokenizer
)as described in thedocs:The tokenizer used to preprocess the data. If provided, will be used to automatically pad the inputs the maximum length when batching inputs, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model.hth!
2021-06-21T17:06:46Z
Hi@lewtun. Thanks very much for that - it works now. I will check out the docs for some further information.Also, I am trying to add special tokens to my model using the code found here:Tokenizer — transformers 2.11.0 documentationHere is the code I am using:# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['🥵', '👏'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizerThe error:NameError                                 Traceback (most recent call last)

<ipython-input-27-203dc3e7172a> in <module>()
      1 # Let's see how to increase the vocabulary of Bert model and tokenizer
----> 2 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
      3 model = BertModel.from_pretrained('bert-base-uncased')
      4 
      5 num_added_toks = tokenizer.add_tokens(['🥵', '👏'])

NameError: name 'BertTokenizer' is not defined
2021-06-22T15:24:32Z
hey@anon58275033what version oftransformersare you using? does the problem go away if you upgrade to the latest version?