Fine-tuning BERT with deterministic masking instead of random masking
I want to fine-tune BERT on a specific dataset. My problem is that I do not want to mask some tokens of my training dataset randomly, but I already have chosen which tokens I want to mask (for certain reasons).To do so, I created a dataset that has two columns:textin which some tokens have been replaced with[MASK](I am aware of the fact that some words could be tokenised with more than one token and I took care of that) andlabelwhere I have the whole text.Now I want to fine-tune a BERT model (say, bert-base-uncased) using Hugging Face’stransformerslibrary, but I do not want to useDataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.2)where the masking is done randomly and I only can control the probability. What can I do?