Ctc bpe conformer overfit
Hi, I'm trying to train a bpe-level conformer and am facing overfitting.

In this regard, I have a few questions and I would be very grateful for the answers to them:
1) Have you encountered overfitting of the BPE conformer and BPE models in general?
2) Have you tried to experiment with char level vs bpe level models? If yes, what are you facing? I mean over/underfitting, stability, approximate increase in metrics.
3) I saw [configs ](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/conformer/conformer_ctc_bpe.yaml) for training conformer from nemo, thanks for posting them, but I couldn't figure out what the global batch size was. [2048](https://github.com/NVIDIA/NeMo/issues/4183#issuecomment-1128444967)?
4) What are some tips for tuning regularization for a conformer?
5) How trainning stability depends on the size of the dictionary?

Before that, I used the char-level conformer, and did not encounter overfitting. I don't have a lot of data, 370 hours for one domain and 130 for another. Therefore, I first train the base model, and then finetune it on domain data. I did some experiments:
1) I finetuned stt_ru_conformer_ctc_large using my domain data and got pretty good quality (better my char level model)
2) I trained my base model, just like I did before with char-level models, and then tried to finetuned it on domain data

The training data configurations are completely the same, except for the weights of models and tokens of tokenizers (the sizes of models and tokenizers are the same).
The length of my audios is from 0.1 to 10 seconds, the size of the bpe dictionary is 128, the average transcription length is 53 characters or 31 tokens.
I use the same (as in nemo models) spec augs, except mask value (mean instead of 0).

I used AdamW, a constant scheduler with lr = 0.0001, global batch size = 512 and faced with overfitting (see plots), then I reduced lr by 10 times and increased global batchs size by 3 times (1536 in total), but this did not solve the problem cardinally. The metrics I get (before the start overfitting) are worse than not only the experiment on finetuning the base model from NeMo, but also worse than the experiments on finetuning my char-level base model.

Plots for char based model:
![newplot (18)](https://github.com/NVIDIA/NeMo/assets/49618087/d32b972b-d27f-4332-9af1-0fc3bb810bc4)
![newplot (15)](https://github.com/NVIDIA/NeMo/assets/49618087/3818469d-2a1c-4b50-a989-75b867915842)

Plots for bpe based model:
![newplot (20)](https://github.com/NVIDIA/NeMo/assets/49618087/6c35247c-fa0b-457e-9675-1222838749e9)
![newplot (19)](https://github.com/NVIDIA/NeMo/assets/49618087/788d546d-b82f-44e4-b015-600df36437df)

@titu1994 

Thank you in advance