BertForMaskedLM model require fine-tuning?
Hi, I am interested in the BertForMaskedLM model, and from the documentation it seems like it could predict the likelihood of a masked token directly from the pretrained BERT model?huggingface.coBERTWe’re on a journey to advance and democratize artificial intelligence through open source and open science.However from looking at the code:github.comhuggingface/transformers/blob/v4.21.1/src/transformers/models/bert/modeling_bert.py#L1307def __init__(self, config):super().__init__(config)if config.is_decoder:logger.warning("If you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for ""bi-directional self-attention.")self.bert = BertModel(config, add_pooling_layer=False)self.cls = BertOnlyMLMHead(config)# Initialize weights and apply final processingself.post_init()def get_output_embeddings(self):return self.cls.predictions.decoderdef set_output_embeddings(self, new_embeddings):self.cls.predictions.decoder = new_embeddingsIt seems that there is an extra LM head that projects a linear layer on top of the output hidden vectors which is then dot-producted with the vocabulary to produce the likelihood. I am wondering how the weights for this head is loaded from (as from_pretrained should only load the weights for the BERT encoder right?) or is it set to some default value each time (I noticed running the model gave the same value each time) and if using BERT in this way to predict the likelihood of a masked token requires pre-training or given the pre-training that BERT goes through would be unnecessary?