Bert Ner finetuning issue
Problem with Bert Ner finetuningHi everyone. I have annotated a corpus with emails and websites and I fine-tuned bert for recognizing them but when I use it, it gives splitted results for example if in the sentence there iswww.example.com, it gives 5 entities of type website (www,.,example,.,com). How can I solve?
2024-05-06T07:43:08Z
Hi,Models like BERT are so-called subword token models, which means that they operate on a (subword) token level, rather than the word level. This means that a world like “hello” might get tokenized into multiple tokens “hel”, “lo”, and then BERT is trained to predict the labels for each individual token.Hence after training, one needs to recombine the labels that BERT predicted at the token level back to the word level. Refer to mydemo notebookwhich includes an inference section at the end.Basically we can leverage the tokenizer as it has a feature called theoffset_mappingwhich contains the character offsets of all the tokens. This section also explains it in more detail:Fast tokenizers’ special powers - Hugging Face NLP Course
2024-05-06T08:52:12Z
Hi,thank you so much for the detailed explanation.I will check it out.Best regards
2024-05-06T20:52:42Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.