Multilingual FastConformer change vocabulary test WER > 1 but validation WER = 0.16
After you finish training the model, you should not call change vocabulary. Change vocabulary is only used when you want to finetune a pretrained model onto a custom dataset or language.  Sorry to clarify, I meant calling change vocabulary before finetuning on my custom dataset. Before finetuning, when I specify the aggregate tokenizer in the config file (shown below) without calling `change_vocabulary` , the test WER valid and below 1. 
```
model:
  ...
  tokenizer:
    type: "agg"  # aggregate tokenizer
    langs:
      en:
        dir: "<path to the directory that contains the tokenizer files>"
        type: "bpe"  # can be "bpe" or "wpe"
      es:
        dir: "<path to the directory that contains the tokenizer files>"
        type: "bpe"  # can be "bpe" or "wpe"
```
However, when I use `change_vocabulary` to replace the decoder with a larger size the test set WER is > 1 although the training is stable.  This is the order of operations 

1) Load a model - that has 1024 tokenizer for English only. Eng wer < 1, does not support other languages v

2) Call change vocab, make it multilingual agg Tokenizer. Eng and all supported languages WER = 1 of higher (untrained decoder)

3) finetune model on new dataset. Wer on eng and all languages < 1. Save Nemo model with save_to. 

4) Restore model using restore_from, all languages should have wer < 1  

--

I'm a bit confused as to where 
```yaml
model:
  ...
  tokenizer:
    type: "agg"  # aggregate tokenizer
    langs:
      en:
        dir: "<path to the directory that contains the tokenizer files>"
        type: "bpe"  # can be "bpe" or "wpe"
      es:
        dir: "<path to the directory that contains the tokenizer files>"
        type: "bpe"  # can be "bpe" or "wpe"
```

Is being used. Configs are only used for scratch training - you don't need to call change vocabulary for such models. 
 Yes I followed the steps that you listed. During the change vocab step I changed the vocab size to 2560 (512 tokens for 5 languages). I restored the model without calling change vocabulary during inference time and WER > 1.

Another point to mention is that the multilingual model seems to be predicting tokens only from 1 language (predictions are in a fixed range e.g. 512-1024) for all languages' test set. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. @choomegan Did you manage to solve this issue? > @choomegan Did you manage to solve this issue?

Hi @ThomasKluiters Nope, I did not manage to find a solution. Do let me know if you have any ideas!
I trained a multilingual (5 language) FastConformer EncDecCTCModel following the tutorial https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Multilang_ASR.ipynb. Training performs as expected and validation WER reaches minimum of 0.16, however when I evaluate the model on different language's test set by loading from either a .ckpt or .nemo file the WER is > 1 for all languages. 

If I exclude the `asr_model.change_vocabulary` step to replace the decoder (as shown below) then the test set WER is all way below 1, and closer to the validation WER. 
```
asr_model.change_vocabulary(
        new_tokenizer_dir=new_tokenizer_cfg,
        new_tokenizer_type="agg",
    )
```

However, I am interested in replacing the decoder as I would like to increase the aggregated tokenizer vocabulary beyond the pretrained FastConformer's 1024 vocabulary size.

Appreciate any help on this issue!