How to use custom positional embedding while fine tuning Bert
Hi, I want to fine-tune Bert for question answering but instead of using the built-in position embeddings (which are the indices of the tokens), I want to add my own positional embedding (for example the sinusoidal positional embedding mentioned in the paper “Attention is all you need”). I understand one way to do that would be to modify the input embeddings in the BertModel class, but I am not sure how to modify that class. If there are better ways, I would be happy to follow. Any suggestions would be highly appreciated.
2022-08-19T07:42:39Z
Did you solve this problem? I’m trying to solve same thing.
2022-09-14T12:53:07Z
All you need is change BertEmbeddings class, you can build a new class and inherit BertEmbeddings.You can follow the post.(I wrote the solution in it.)How to use additional input features for NER?BeginnersHi f3n1Xx, I’m still stuck at the same problem and it’s also for a biological problem. There is no code in the link you provided. Would you mind posting here how you added additional features. If you can’t release the entire code could you just show a simplified example please? Thank you!