What tokenizer.as_target_tokenizer() used for?
Hi, as the title said. I just don’t understand the use-cases of this context managerwith tokenizer.as_target_tokenizer():I have read the doc thoughTHANKS
2023-12-28T03:19:29Z
according tounderstand differences in tokenization · Issue #18455 · huggingface/transformers · GitHubthis context manager is deprecated so don’t worry