Bert ner classifier
hi,I fine-tune the bert on NER task, and huggingface add a linear classifier on the top of model. I want to know more details about classifier architecture. e.g. fully connected + softmax…thank you for your help
2021-04-29T15:47:10Z
Hi! Can you be a little bit more specific about your query?Just to give you a head start,In general, NER is a sequence labeling (a.k.a token classification) problem.The additional stuff you may have to consider for NER is, for a word that is divided into multiple tokens by bpe or sentencepiece like model, you use the first token as your reference token that you want to predict. Since all the tokens are connected via self-attention you won’t have problem not predicting the rest of the bpe tokens of a word. In PyTorch, you can ignorecomputing loss(seeignore_indexargument) of those tokens by providing -100 as a label to those tokens (life is so easy with pytorch).Apart from that, I didn’t find any more additional complexity in the training NER model.Some other implementation details you need to check,One important Note: So far I remember (please verify), In conll, german or dutch dataset there are 2-3 long sentences in the test dataset. Sequence labeling doesn’t work like sentiment analysis. You need to make sure your sentence is not cut down by the max_sequence_len argument of the Language Model’s tokenizer. Otherwise, you will see a little bit of discrepency in your test F1 sore. An easier hack for this problem is to divide the sentence into smaller parts and predict them one by one and finally merge them.Imo Self-attention and CRF layer is theoretically different but in application some of the problem that CRF solved in prior model, self-attention can also solve them (because they create a fully connected graph). So using softmax is more preferable than a CRF layer.The score that the originalBERT paperreported are not reproducible and comparable with most of the papers since they used document level NER fine-tuning.If you still have query about the architecture you can follow this,Guillaume Genthial blog – 5 Apr 17Sequence Tagging with TensorflowGloVe + character embeddings + bi-LSTM + CRF for Sequence Tagging (Named Entity Recognition, NER, POS) - NLP example of bidirectionnal RNN and CRF in Tensorflowyou only have to replace hierarchical rnn with transformer as the encoder.You can check the following paper’s for more info,Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT - ACL Anthologyhttps://arxiv.org/pdf/2007.07683.pdfhttps://arxiv.org/pdf/2004.12440.pdfhttps://arxiv.org/pdf/2004.13240.pdf(beware, this is my publication. I may be biased, still not accepted)Please let me know if you have more queries.
2021-04-30T02:57:34Z
Thank you very much for your explanation. It let me learning a lot.I print my model ,then I find it has a classifier. I want to know what is the architecture.擷取.PNG706×346 62.7 KB
2021-04-30T03:14:22Z
As you can see, the classifier is a single dense layer.It is probably pointing out from here if you are usingBertForSequenceClassification,transformers/modeling_bert.py at b29eb247d39b56d903ea36c4f6c272a7bb0c0b4c · huggingface/transformers · GitHubIf you are usingBertForTokenClassification, it is pointing out here,github.comhuggingface/transformers/blob/b29eb247d39b56d903ea36c4f6c272a7bb0c0b4c/src/transformers/models/bert/modeling_bert.py#L1648class BertForTokenClassification(BertPreTrainedModel):_keys_to_ignore_on_load_unexpected = [r"pooler"]def __init__(self, config):super().__init__(config)self.num_labels = config.num_labelsself.bert = BertModel(config, add_pooling_layer=False)self.dropout = nn.Dropout(config.hidden_dropout_prob)self.classifier = nn.Linear(config.hidden_size, config.num_labels)self.init_weights()@add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))@add_code_sample_docstrings(tokenizer_class=_TOKENIZER_FOR_DOC,checkpoint=_CHECKPOINT_FOR_DOC,output_type=TokenClassifierOutput,config_class=_CONFIG_FOR_DOC,)For setting up num_of_label, please change the variable in your config file.
2021-05-03T07:21:08Z
hi,can i think self.classifier = nn.Linear(config.hidden_size, config.num_labels) as a fully-connected layer.input dimension is config.hidden_size and out dinension is config.num_labels. as showntfdl_0401840×901 114 KB
2021-05-03T11:54:43Z
yes, this is just a linear layer.