Replacing last layer of a fine-tuned model to use different set of labels
I’m trying to fine-tunedslim/bert-base-NERusing thewnut_17dataset.Since the number of NER labels is different, I manually replaced these parameters in the model to get rid of thesize mismatch error:model.config.id2label = my_id2labelmodel.config.label2id = my_label2idmodel.config._num_labels = len(my_id2label) ## replacing 9 by 13However, when training starts I get the following error which I don’t know how to handle:Expected input batch_size (1456) to match target batch_size (1008).Has anyone handled this manually?@sgugger@phosseiniWon’t it be great if we can have a solid function that handles the head replacements for fine-tuning.Shapes:tokenized_wnut[‘train’].shape = (3394, 7)tokenized_wnut[‘validation’].shape = (1009, 7)Model config after “manual” modifications:BertConfig {
  "_name_or_path": "dslim/bert-base-NER",
  "_num_labels": 13,
  "architectures": [
    "BertForTokenClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "O",
    "1": "B-corporation",
    "2": "I-corporation",
    "3": "B-creative-work",
    "4": "I-creative-work",
    "5": "B-group",
    "6": "I-group",
    "7": "B-location",
    "8": "I-location",
    "9": "B-person",
    "10": "I-person",
    "11": "B-product",
    "12": "I-product"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "B-corporation": 1,
    "B-creative-work": 3,
    "B-group": 5,
    "B-location": 7,
    "B-person": 9,
    "B-product": 11,
    "I-corporation": 2,
    "I-creative-work": 4,
    "I-group": 6,
    "I-location": 8,
    "I-person": 10,
    "I-product": 12,
    "O": 0
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.14.1",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 28996
}
2021-12-21T13:55:26Z
Hi,Seethis postthat will solve your issue
2021-12-21T19:15:45Z
Thanks, but apparently it’s not completely resolved. The solutions suggested don’t work for my problem.
2021-12-22T09:39:00Z
it’s not completely resolved.=> can you clarify? The error that you’re getting:Expected input batch_size (1456) to match target batch_size (1008).suggests that it has to do with the preparation of the dataset, rather than the last layer of the model.
2021-12-23T00:57:53Z
Thank you@nielsrfor being responsive.That error is resolved now, but the question is"does simply changing the number of labels mean that we have changed the classifier head?!"By the way, for my problem, I had to do these modifications:model_name = "dslim/bert-base-NER"
mymodel = AutoModelForTokenClassification.from_pretrained(model_name, num_labels=len(my_id2label), ignore_mismatched_sizes=True)
...
mymodel.config.id2label = my_id2label
mymodel.config.label2id = my_label2id
mymodel.config._num_labels = len(my_id2label) ## replacing 9 by 13
mymodel.config.num_labels = len(my_id2label)
2021-12-23T09:43:18Z
Aliseyfi:“does simply changing the number of labels mean that we have changed the classifier head?!”=> yes. If you change the number of output neurons, then you’ll get a new linear layer whose weights and bias are randomly initialized.Also, not sure why you have_num_labels, you should only havenum_labels.
2021-12-23T16:59:30Z
Thanks for double confirming@nielsr.nielsr:yes. If you change the number of output neurons, then you’ll get a new linear layer whose weights and bias are randomly initialized._num_labelsis the 2nd parameter in Bert Config (shown in the model config above), and yes, changing it is optional.