Finetuning Citrinet using speech_to_text_bpe poor WER
**Describe the bug**

I'm trying to use speech_to_text_bpe.py to fine-train Citrinet. I've managed to fine-train citrinet with reasonable success using my own code getting a WER of 16 on 8Khz audio upsampled to 16Khz. However, I would rather use the scripts if possible as it will be easier to maintain longer term. When I use the speech_to_text_xxx.py scripts, the results are poor with my WER sitting at around 95, so clearly I'm doing something wrong. 

**Steps/Code to reproduce bug**

I first create by subword tokenizer, this was using spe but i've also tried wpe. To be honest I'm not sure about the correct setting for vocab_size and if 'spe' is the correct tokenizer to use.

!python /root/asr16000/NeMo/scripts/tokenizers/process_asr_text_tokenizer.py \
  --manifest="/root/asr16000/training/training_samples.json" \
  --data_root="/root/asr16000/tokenizer" \
  --vocab_size=64 \
  --tokenizer='spe' \
  --no_lower_case \
  --spe_type='unigram' \
  --spe_character_coverage=1.0 \
  --log

Then I train the model, I assume I should use the '_bpe' script version here.

!python /root/asr16000/NeMo/examples/asr/speech_to_text_bpe.py \
    --config-path='/root/asr16000/NeMo/examples/asr/conf/citrinet/' \
    --config-name='citrinet_512' \
    model.train_ds.manifest_filepath='/root/asr16000/training/training_samples.json' \
    model.train_ds.batch_size=16 \
    model.validation_ds.manifest_filepath='/root/asr16000/training/eval_samples.json' \
    model.validation_ds.batch_size=16 \
    model.test_ds.manifest_filepath='/root/asr16000/training/eval_samples.json' \
    model.test_ds.batch_size=16 \
    model.tokenizer.dir='/root/asr16000/tokenizer/tokenizer_spe_unigram_v64/' \
    model.tokenizer.type='bpe' \
    model.optim.name="adamw" \
    model.optim.lr=0.001 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    model.optim.sched.warmup_steps=2000 \
    trainer.gpus=1 \
    trainer.accelerator="ddp" \
    trainer.max_epochs=100 \
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name="nemo-asr-script-citrinet" \
    exp_manager.wandb_logger_kwargs.project="speech_to_text" 

The /root/asr16000/NeMo directory is a clone of the 1.2 nemo repo. I have about 350 hours of audio for training.

**Expected behavior**

I would expect the WER to be better than 95, I was hoping for something lower than 16. Any pointers would be much appreciated.

**Environment overview (please complete the following information)**

 - Environment location: AWS Sage Maker
 - Method of NeMo install: %pip install nemo_toolkit['all']==1.2.0

**Environment details**

- OS version : Debian GNU/Linux 10 (buster)
- PyTorch version: 1.9.0+cu102
- Python version: 3.6.13

**Additional context**
GPU model A100
