Regarding the training details of the ASR model on limited resources
Hi, @titu1994.
I am trying to train a conformer_ctc_bpe (small) model with librispeech960 using four NVIDIA 2080 GPUs (11GB memory).
But, when training model from scratch, I can't reproduce the reported performance with my experimental settings.


Looking at your training details, it seems that a batch size of 2K (maybe 2,048?) and a learning rate of 5.0 are used.

However, with my GPUs of 11GB, I can't use such a large batch size; the maximum batch size I can use is 12.

Could you please share some guidelines and tips about appropriate training details for my experimental condition..?
(i.e., **batch size**, **learning rate**, **warmup step**, **weight decay**, **accumulate_grad_batches**, etc. or another optimizer / scheduler ..?)

Some configurations I set are as follows
```
model:
  sample_rate: 16000
  log_prediction: true
  ctc_reduction: 'mean_batch'
  skip_nan_grad: false
  
  train_ds:
    manifest_filepath: "/home/shmun/NeMo/data/train_all_960.json"
    sample_rate: 16000
    batch_size: 12 # you may increase batch_size if your memory allows
    shuffle: true
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    trim_silence: false
    max_duration: 16.7
    min_duration: 0.1
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    bucketing_strategy: "synced_randomized"
    bucketing_batch_size: null

  tokenizer:
    dir: "/home/shmun/NeMo/data/tokenizer_ctc_conformer_large_128/"
    type: "bpe"

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2 
    time_masks: 5  #10
    freq_width: 27
    time_width: 0.05

  optim:
    name: adamw
    lr: 5.0  #2.0
    betas: [0.9, 0.98]
    weight_decay: 1e-3

    sched:
      name: NoamAnnealing
      d_model: 176
      warmup_steps: 10000
      warmup_ratio: null
      min_lr: 1e-6
```