Can I add custom truncation options to use special token?
Hello I’m going to use the truncation function in my tokenizers.I know that huggingface cut it tomax_lengthoption,but I want to cut it not only usingmax_length, also using the token I set.ex)To be, or [DIV] not to be, that is the question→To be, ornot to be, that is the questionLike this! how can I add it?