Calculate F1 score in a NER task with BERT
Hi everyone,I fine tuned a BERT model to perform a NER task using a BILUO scheme and I have to calculate F1 score.However, in named-entity recognition, f1 score is calculated per entity, not token.Moreover, there is the Word-Piece “problem” and the BILUO format, so I should:aggregate the subwords in wordsremove the prefixes “B-”, “I-”, “L-” from each entitycalculate the F1 score on the entityBefore I spend hours (if not days) to try to implement such code, I would like to know if an implemented solution already exists.Thanks in advance
2021-01-15T14:21:26Z
You should use thedatasetsmetricseqevalthat will do all of this for you. Check thenewrun_nerscriptfor an example.
2021-01-15T15:51:09Z
Thanks for the hint@sgugger.I have a question.seqevalofdatasetsis the same implementation ofthis?If yes, in my case this is a problem, because F1 score with BILOU format can be calculated only in strict mode, while I need the default one.
2021-01-15T17:11:08Z
Ah yes, it’s the same so it won’t be useful to your use case, sorry.