Adding new tokens to pretrained asr vocab 
Is it possible to add new tokens to the tokeniser of a pretrained model. say I want to fine tune on some new data, which contains few new tokens eg. punct or nums can I add these to the decoder vocabulary?

Here I don't want to change the entire vocabulary so the initial decoder weights will remain the same hence I can reuse them, but for the new tokens the decoder weights can be randomly initialised and learned during fine-tuning 

The only other way I knew to add new tokens was to change the entire tokeniser. but then I am unable to use the decoder. 