Why sep_token_id is same as eos_token_id for allenai/led-base-16384
I am wondering whysep_token_idis same aseos_token_idforallenai/led-base-16384Huggingface model. It creates a confusion during inference whether</s>token refers toeos_tokenorsep_token, both ideally would be used in different places in text generation.Steps to reproduce the issue:from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384")
model = AutoModelForSeq2SeqLM.from_pretrained("allenai/led-base-16384")
print(tokenizer.eos_token_id) # prints 2
print(tokenizer.sep_token_id) # prints 2 again