AutoTokenizer vs. regular Tokenizer
I understand that there are a few different tokenizers; e.g., DistilBertTokenizerFast, etc. However, I don’t understand the concept of “Auto” in tokenizer selection. Using IMDB text as an example. What do I get if I use AutoTokenizer to tokenize the text?
2021-06-01T13:02:53Z
TheAutoTokenizerwill work on any checkpoint and pick the proper architecture for you (whereasDistilBNertTokenizerFastwill only work for distilbert checkpoints).
2021-06-02T17:30:58Z
Thanks. In this case, are there reasons to use DistilBertTokenizerFast?