Keep encountering the KeyError message while trainer.fit()
**Describe the bug**

Hi, thanks for providing such a helpful toolkit. I'm currently following the tutorial from [Nvidia-examples](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_with_NeMo.ipynb#scrollTo=3FT0klSV268p), and applying my own Mandarin Chinese corpus. Since our dataset is Mandarin Chinese, the only different setting is the 'labels' in the configuration files. Thus, I directly change the labels through the command below:
`params['model']['train_ds']['labels'] = [' ']+list(unique_char)+params['labels']`
`params['labels'] = [' ']+list(unique_char)+params['labels']`
where `unique_char` is a set that includes all the unique characters in our train_manifest.

p.s. I preprocess all the texts of each audio to a single character form as below:
`{audio_filepath: example.wav, duration: 15, text: '你 好 嗎'}`
Hence, aggregating all the text information from train_manifest to form `unique_char` should include all the characters.

However, I still met similar error messages while doing trainer.fit() as below:
```
  File "/opt/conda/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/collections/asr/metrics/wer.py", line 1074, in decode_ids_to_tokens
    token_list = [self.labels_map[c] for c in tokens if c != self.blank_id]
  File "/opt/conda/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/collections/asr/metrics/wer.py", line 1074, in <listcomp>
    token_list = [self.labels_map[c] for c in tokens if c != self.blank_id]
KeyError: 1669
```
BTW, it would run several iterations in Epoch 0 to pop out this error message.
Is there anyone facing similar questions that can help me fix the problem?
Appreciate your help and time, thank you!