How to make GPT2 Tokenizer actually add special tokens
I’m using aGPT2TokenizerFasttokenizer. When tokenizing, the tokenizer will not add special tokens, even whenadd_special_tokens=True. This is baffling to me, but appears to beintended behavior. How can I convert this tokenizer to one that does the exact same thing but actually adds special tokens?
2023-09-16T15:15:12Z
maybe you need to explain more specific. like how do code implement the add special token?
2023-09-16T17:12:58Z
These are just standard special tokens, in this casebosandeostokens. They are held in thespecial_tokens_maptokenizer attribute. When saved, they end up in thespecial_tokens_map.jsonfile andvocab.jsonfile. I’m not doing anything unusual or custom.The issue is with other tokenizers, tokenizing withadd_special_tokens=Truecausesbosandeostokens to be added automatically, but withGPT2TokenizerFastthis is ignored.GPT2TokenizerFastbehavior requires manually addingbosandeostokens to text strings before tokenizing and numericalizing. This has caused some very sneaky bugs to crop up because theadd_special_tokensbehavior is not documented anywhere.One thing I’ve done is created aRobertaTokenizerFastfrom thevocab.jsonandmerges.txtfrom a savedGPT2TokenizerFast. The Roberta type tokenizer correctly addsbosandeostokens whenadd_special_tokens=True, but this feels hacky and confusing when we use this tokenizer with GPT2 models.I’m looking for a way to getGPT2TokenizerFastto addbosandeostokens whenadd_special_tokens=True, which is the standard behavior of other tokenizers in the library.
2024-02-08T07:54:35Z
Here running into the same. Is there a way to automatically add thebos_tokenandeos_token?