Does tokenizer changed during model training
I want to know during model training, does tokenizer alter it self, (i mean does it train) or it remain the same as original one.For example i used theAutoTokenizer.from_pretrained(model_name,local_files_only=False)after model training, is there any change to tokenizer?
2022-06-16T01:44:22Z
Hi, Talha,Tokenizer id does not change. For example, the token_id of[['sep']]is 102. It will be 102 through the training. But the embedding hidden state will be changed.
2022-08-11T13:47:47Z
One doubt. So lets say I have trained two models of “bert-base-uncased” to do different classification tasks(same number of classes).So, now, can I use the tokenizer of one model1 to tokenize the other one? Will the result be different compared to using two different tokenizers?