[Tokenizers]What this max_length number?
When I called FastTokenizer, I could see the strange number of “model_max_length” as “1000000000000000019884624838656”. What is the meaning of the strange model max length?from transformers import AutoTokenizer
model_name = 'microsoft/mdeberta-v3-base'

tokenizer = AutoTokenizer.from_pretrained(model_name)
vars(tokenizer)스크린샷 2022-12-27 오전 11.01.302772×1272 359 KB
2022-12-27T07:19:44Z
It’s just the largest integer in this precision, because this model does not have a max length.
2024-01-18T23:32:50Z
fyi this can happen for llama2-7b.