No Convergence on SqueezeFormer character vocab based CTC Models
I have been trying to train the SqueezeFormer-based ASR model for the Hindi language. Model is only converging if I use BPE based vocabulary while if I used Non BPE based vocabulary on same training data model fails to converge.

Below are the experiment logs.
**BPE-BASED VOCABULARY**
![bpe](https://user-images.githubusercontent.com/18329235/215416244-fb4a23c3-cced-418d-b854-5179269f0911.png)

**NON BPE BASED VOCABULARY**
![non-bpe](https://user-images.githubusercontent.com/18329235/215416329-4b767c12-d4a7-49e5-aaa8-43205755fc46.png)

**MODEL DETAILS**
I am training SqueezeFormer Medium model with the following configurations as provided in the config file in the repo.
 
|Vocab type| Model        | d_model | n_layers | n_heads | time_masks | lr     | time_reduce_idx | GBS  |
|-------------|--------------|---------|----------|---------|------------|--------|-----------------|------|
|BPE         | Medium       | 324     | 20       | 4       | 7          | 1.5e-3 | 9               | 1024 |
|Non BPE| Medium       | 324     | 20       | 4       | 7          | 1.5e-3 | 9               | 1024 |

**Expected behavior**
Both model should be converged to nearly similar point.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal]
 - Method of NeMo install: [pip install or from source]
 `conda create --name nemo python==3.8`
 `conda activate nemo`
 `pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html`
 `apt-get update && apt-get install -y libsndfile1 ffmpeg`
 `pip install Cython`
 `pip install nemo_toolkit['all']`

**Environment details**
Otherwise, please provide:
- OS version: ubuntu 20.04
- PyTorch version : 1.10.1+cu111
- Python version: Python 3.8.0


Is there some configuration issue with bpe vs non bpe based model ??
