The reference output is not correct when fine-tuning STT Zh Citrinet1024 Gamma 0.25 Model
**Describe the bug**

> my dataset situation:
I only have a dataset with 3Hs of my target accent and I have 2H for train+dev and 1H for valid, and the charset size of it is very small with only 394(It means I have multiple people speaking same lines) (I don't know if this is what causing the problem: poor dataset size and not adequate enough charset)

The reference output from the runtime is not correct.
Following This issue: https://github.com/NVIDIA/NeMo/issues/2725#issue-978965808, I freeze enc. and dec..
But "After several epoch training ,The output reference and predicted are correct" didn't happen.
```bash
# This is just a bunch of meaningless characters, even has some alphabets in it.
[NeMo I 2022-10-22 03:38:50 wer:985] reference:仑兆书偻使仿侬份侣交仑兆书偻使井交X亩且亥亟使佳仰但东伴傀使似侥乘Z使佳
[NeMo I 2022-10-22 03:38:50 wer:986] predicted:难道联美不折者物了明女张找您美的部累吗突的端头在不仅有天离而铁来姑勇华偏而O军
```

```bash
# While following the tutorial step by step can have a correct one on jp..
# Means as an (cultural?) entrance to Japan, the standard Japanese is needed.
# (Don't they just be decoded from unicode in a same way?)
[NeMo I 2022-10-21 09:30:32 wer:985] reference:日本のエスヘラントとしてやはり標準語は必要た
[NeMo I 2022-10-21 09:30:32 wer:986] predicted:郵奈郵奈ょ奈ょ彼加奈施圧息鯨会加会加襞会を圧を圧を赤っ紅奈施奈素米調っ出航米埋裏似樫葉襞浜民た霧営端所坊事備出を樫赤複容紅竹郵沈繊越な素奈所陸霧幼端樫輪み米よ備霧な樫出樫腹飛備紅狩紅樫出素宿敗街暮排樫赤備っな宿敗杯備焦樫要彼事匠リ素赤要宿邦よ樫排を樫紅持紅街備所戻米所解え狩彼演属会住街圧策赤拠防米飛備霧を米狩米赤備複樫紅樫の紅樫航竹戻父所坊証郵拠な複な宿坊飛陸含な樫郵赤彼赤ん赤
```

And my Learning rate drops very fast while train_cer and valid_cer is still very high.(around 1...)

**Steps/Code to reproduce bug**
1. follow the [lang. finetune tutorial](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb#scrollTo=DoWNVNYGOaMX), but doing fine-tuning on zh citrinet with bigger kernel size, skip oov removal, skip argumentation.(Character wise, not the sub-word method in part B)

2. convert the code in notebook to a script and run the script.

> I have my code here.  [run.txt](https://github.com/NVIDIA/NeMo/files/9842185/run.txt)


**Expected behavior**

Supposed to have a readable reference.(And when use the trained model to do transcription, it threw up same sort of meaningless codes too.)

**Environment overview**

 - Environment location: Bare-metal
 - Method of NeMo install: from source(clone and using the reinstall script)

**Environment details**

- OS version Ubuntu 18
- PyTorch version 1.12.1
- Python version 3.8

**Additional context**

Add any other context about the problem here.
Example: GPU model V100 32G x2
