Why aren't all weights of BertForPreTraining initialized from the model checkpoint?
When I load aBertForPretrainingwith pretrained weights withmodel_pretrain = BertForPreTraining.from_pretrained('bert-base-uncased')I get the following warning:Some weights of BertForPreTraining were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['cls.predictions.decoder.bias']Why aren’t all the weights incls.predictionsinitialized from the saved checkpoint?The model seems to produce reliable token prediction outputs (without further training). In particular, it produces the same outputs as a model loaded withmodel_masked = BertForMaskedLM.from_pretrained('bert-base-uncased')Here’s code verifying this in an example:s = ("Pop superstar Shakira says she was the [MASK] of a random [MASK] by a [MASK] "
     "of [MASK] boars while walking in a [MASK] in Barcelona with her eight-year-old "
     "[MASK].")
inputs = tokenizer(s, return_tensors='pt')

outputs_pretrain = model_pretrain(**inputs)
outputs_masked = model_masked(**inputs)
assert torch.allclose(outputs_pretrain["prediction_logits"], outputs_masked["logits"])Incidentally, when loadingmodel_masked, I don’t get a warning about newly initialized weights incls.predictions. All newly initialized weights are incls.seq_relationship, which is reasonable since if we only care about masked LM, the information from the base model regarding next sentence prediction can be safely thrown away.
2021-10-05T07:23:56Z
mgreenbe:BertForPreTrainingTheBertForPreTrainingmodel is BERT with 2 heads on top (the ones used for pre-training BERT, namely next sentence prediction and masked language modeling). Thebert-base-uncasedcheckpoint on thehubonly includes the language modeling head (it’s actually suited to be loaded into aBertForMaskedLMmodel). You can also see this in the config filehere.
2021-10-05T10:27:59Z
Has the NSP head ever been included? I.e. are those weights available anywhere?
2021-10-05T14:48:30Z
But aren’t the weights for the LM head stored oncls.predictions? If so, shouldn’tcls.predictions.decoder.biasbe initialized from the checkpoint? The weights stored oncls.seq_relationshipshould be the randomly initialized ones, no?