Is "Some weights of the model were not used" warning normal when pre-trained BERT only by MLM
Hello guys,I’ve trained BERT model from the scratch using BertForMaskedLM and trainers. When I use AutoModelForSequenceClassification to fine-tune my model for a text classification task, I get a warrning about weights initialization. Is it normal to get a warning such as in the below or am I doing something wrong ?Some weights of the model checkpoint at ./cased/bert-wikidump-50mb-mlm/model were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./cased/bert-wikidump-50mb-mlm/model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias', 'classifier.weight', 'classifier.bias']Loading pre-trained model with AutoModelForSequenceClassificationfrom transformers import AutoModelForSequenceClassification, AdamW, AutoConfig
config = AutoConfig.from_pretrained(PATHS["model"]["cased"]["local"], num_labels=df.category.unique().size)

model = AutoModelForSequenceClassification.from_pretrained(PATHS["model"]["cased"]["local"], config=config)Code for training BERT from scratch with only MLM taskfrom transformers import BertConfig
config = BertConfig(vocab_size=64_000)

from transformers import BertForMaskedLM
model = BertForMaskedLM(config=config)

from transformers import Trainer, TrainingArguments

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=bert_cased_tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir=PATHS["model"]["cased"]["training"]["local"],
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_gpu_train_batch_size= 8, ## 512 max sequence lenght, 64 sequence count
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
2021-04-22T02:22:33Z
Yes, the warning is telling you that some weights were randomly initialized (here you classification head), which is normal since you are instantiating a pretrained model for a different task. It’s there to remind you to finetune your model (it’s not usable for inference directly).
2021-04-22T05:54:38Z
Thanks@sgugger!
2022-04-19T19:12:17Z
Hi@sgugger, may I know how to suppress this information?  I load pretrained model and  multiprocess this , so this notification is visually  overwhelming. Thank you!
2023-08-16T18:32:36Z
Is there a way to avoid some weights to be initialized randomly since already initialized ?
2023-12-15T16:26:27Z
To suppress this output,nlp - Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel - Stack Overflowsuggests to change the verbosity level oftransformers.logging.
2024-03-28T16:22:30Z
Is there a way to use distilbert-base-uncased and other models out-of-the-box without fine tuning to benchmark?