Version comparison 1.0.2 x 0.10.1
Hi

I've been doing transcription training for the Portuguese language for a few months and so far I'm getting great results. But I came across a question when performing a comparison of results from version 0.10.1 and 1.0.2 with the same dataset.
For this comparison I respectively used the scripts:
**(Version 0.10.1)** 
Container https://ngc.nvidia.com/catalog/containers/nvidia:nemo_asr_app_img and notebook 'English to Spanish Cross Language Finetuning'
**(Version 1.0.2)** 
Notebook https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb

In their respective parameters I used the recommended values:
**(Version 0.10.1)**
```
# training parameters 
project.set_am_num_gpus(1)
project.set_am_batch_size(42)
project.set_am_num_epochs(50) # as an example we use 1 epoch but in reality we finetuned the model for 256 epochs
project.set_am_learning_rate(0.0015) # lr in finetuning an order of magnitude smaller than training from scratch
project.manifest.am.train_params.amp_opt_level = 'O1'

# optimizer novograd
project.manifest.am.train_params.beta1 = 0.95
project.manifest.am.train_params.beta2 = 0.25 

project.manifest.am.train_params.lr_policy = "CosineAnnealing"
project.manifest.am.train_params.weight_decay = 0.002
project.manifest.am.train_params.warmup_ratio = 0.12
```

**(Version 1.0.2)**
```
# Setup train, validation, test configs
with open_dict(cfg):    
  # Train dataset  (Concatenate train manifest cleaned and dev manifest cleaned)
  cfg.train_ds.manifest_filepath = train_manifest_cleaned
  cfg.train_ds.labels = list(train_dev_set)
  cfg.train_ds.normalize_transcripts = False
  cfg.train_ds.batch_size = 28
  cfg.train_ds.num_workers = 8
  cfg.train_ds.pin_memory = True
  cfg.train_ds.trim_silence = True

  # Validation dataset  (Use test dataset as validation, since we train using train + dev)
  cfg.validation_ds.manifest_filepath = test_manifest_cleaned
  cfg.validation_ds.labels = list(train_dev_set)
  cfg.validation_ds.normalize_transcripts = False
  cfg.validation_ds.batch_size = 8
  cfg.validation_ds.num_workers = 1
  cfg.validation_ds.pin_memory = True
  cfg.validation_ds.trim_silence = True

with open_dict(char_model.cfg.optim):
  char_model.cfg.optim.lr = 0.01
  char_model.cfg.optim.betas = [0.95, 0.5]  # from paper
  char_model.cfg.optim.weight_decay = 0.001  # Original weight decay
  char_model.cfg.optim.sched.warmup_steps = None  # Remove default number of steps of warmup
  char_model.cfg.optim.sched.warmup_ratio = 0.05  # 5 % warmup
  char_model.cfg.optim.sched.min_lr = 1e-5
```

For all values I'm following the recommendations of the scripts themselves. The only change was that BatchSize could only use 28 in the most recent script and 42 in the oldest script. (That's really a question, does the latest version of NeMo use more VRam? For these tests I'm using an Nvidia T4 - 16GB).
After doing the training on both scripts, I noticed that the older version has 30% better accuracy. Is there any explanation for this? Could BatchSize be the reason?