Conformer training from scratch not achieving the reported WER
Hello! Thanks for providing such a nice toolkit.

I am currently training conformer-small on librispeech-960hr from scratch, following the training recipe and instructions in `examples/asr/conf/conformer/conformer_ctc_bpe.yaml`.
However, I'm not able to reproduce the WER reported in this [webpage](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_ctc_small_ls): reported ones are 3.4/8.8, but our WER is only around 4.5/11 on the clean/other datasets with ~300 epochs of training and doesn't seem to decrease anymore.

Here is the list of hyperparameters I'm using:
* d_model, n_heads, n_layers: 176, 16, 16 
* tokenizer: WPE with vocab_size 128 (generated using the script in NeMo)
* spec_augment: freq_masks 2, time_masks 5, freq_width:27, time_width: 0.05
* lr: 5.0, betas: [0.9, 0.98], weight_decay: 1e-3 (adamw)
* warmup_steps: 10000 (noam annealing)
* Batch size: 2048 (using 8 machines, local batch size 16, and accumulate_grad_batches 16, multiplying up to 2048)
All the other hyperparameters are the same as the default.


Here is one of the learning curves (validation WER on dev-clean) from our runs:
<img width="1665" alt="Screen Shot 2021-12-05 at 1 18 53 PM" src="https://user-images.githubusercontent.com/50283958/144764357-9cc2609c-cf8e-4158-8693-5e6a7b7c5bd8.png">

I'm wondering if we should change the hyperparameters to achieve the reported WER values, or if the 300-epoch of training is not enough for the model convergence, or if there's something that we are missing.
Any clues regarding the hyperparameters, the number of training epochs, etc. would be really appreciated!

Thank you.