Why BPE encoding trained on English and applied on Bengali doesnot return unknown tokens?
I use roberta-base tokenizertokenizer = RobertaTokenizerFast.from_pretrained('roberta-base',add_prefix_space=True)trained on english data to tokenize bengali just to see how it behaves . When I try to to encode a bengali charactertokenizer.encode('বা'), I get[0, 1437, 35861, 11582, 35861, 4726, 2]which means that it finds some tokens in it vocabulary which match bengali characters even though train on english. On further exploration I find these are all special characters['<s>', 'Ġ', 'à¦', '¬', 'à¦', '¾', '</s>']. My question is why does it happen, isn’t it supposed to output unknown tokens when applied on a new language ? Any help greatly appreciated
2024-02-25T16:07:09Z
Hey@Sam2021did you find anything on why the tokenizer behaves like this? Would love to know if you have an update on this