Using a dataset with already masked tokens
I am trying to fine tune BERT for Masked Language Modeling and I would like to use a dataset that already contains masked tokens (I want to mask particular words rather than randomly chosen ones).How can I do this?I am following thesehttps://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb#scrollTo=KDBi0reX3l_ginstructions, but I am not sure which parts of the code I need to change for it to be compatible with a dataset that already has [MASK] tokens in it.Thanks!
2021-01-29T19:20:32Z
The masking is done by the data collatorDataCollatorForLanguageModeling. Just pass alongmlm=Falseto that data collator to deactivate the random masking there.
2021-02-03T16:47:22Z
Thank you so much!