Can I use a Sentencepiece tokenizer model from my ASR model in order to train a NeuralRescorer?
Closing this issue as it was already answered here:

https://github.com/NVIDIA/NeMo/issues/4390
Hi,

I would like to train a Neural Rescorer model for ASR using the transformer_lm.py script. I have one small question:

In the documentation it says: "Our LMs support all tokenizers available in NeMo, but require special beginning-of-string <bos> and end-of-string <eos> tokens." 

But I imagine that the tokenizer of the TransformerLM must be the same as the tokenizer of the ASR model, right? Can I pass the tokenizer model of the ASR model directly to TransformerLM? Should I add a <bos> and <eos> token and change the size of the tokenizer? ( but I imagine the ASR and Transformer should have the same tokenizer size)

Could you help me on this matter?

Thank you in advance!

