Is there any difference in the tokenized output if I load the tokenizer from a different pretrained model
So let’s say if I doGPT2TokenizerFast.from_pretrained('gpt2-medium')vsGPT2TokenizerFast.from_pretrained('distilgpt2')Is there actually any differences in their tokenized output?
2020-08-26T16:30:30Z
In that particular case, I don’t think so, but there are definitely cases where tokenizers from the same model type but different pretrained configurations are different.bert-base-uncasedvsbert-base-casedwould be one clear example.
2020-09-03T07:11:27Z
Thank you for your clarification