A little confusion about learning rate decay parameters for fastpitch and hifigan
Thanks for the continual development and maintenance of this code. 
After modifying min_lr and max_epoch from default, I trained fastpitch for 500 epochs and hifigan for 52 epochs on my dataset, which consists of 25,584 audio clips totaling 44 hours. I found no decay in the learning rate.

fastpitch:
![fastpitch](https://github.com/NVIDIA/NeMo/assets/29267562/3bcd562f-2c20-4c47-89a9-ef42001b8bb2)
```
  optim:
    name: adamw
    lr: 1e-3
    betas: [0.9, 0.999]
    weight_decay: 1e-6

    sched:
      name: NoamAnnealing
      warmup_steps: 1000
      last_epoch: -1
      d_model: 1
      min_lr: 2e-4   #modified,default N/A

trainer:
  num_nodes: 1
  devices: 1
  accelerator: gpu
  strategy: null    #modified,default ddp
  precision: 16
  max_epochs: 500    #modified,default 1000
  accumulate_grad_batches: 1
  # gradient_clip_val: 1000.0
  enable_checkpointing: False
  logger: false
  log_every_n_steps: 5
  check_val_every_n_epoch: 1
  benchmark: false
  num_sanity_val_steps: 0
```

hifigan:
![hifigan](https://github.com/NVIDIA/NeMo/assets/29267562/6d22a307-d91a-4f0f-9c44-1fb8c2f217d5)
```
  optim:
    _target_: torch.optim.AdamW
    lr: 0.0002
    betas: [0.8, 0.99]

    sched:
      name: CosineAnnealing
      min_lr: 1e-5
      warmup_ratio: 0.02

  max_steps: 2500000
  l1_loss_factor: 45
  denoise_strength: 0.0025

trainer:
  num_nodes: 1
  devices: 1
  accelerator: gpu
  strategy: ddp
  precision: 32
  max_steps: ${model.max_steps}
  accumulate_grad_batches: 1
  enable_checkpointing: False  # Provided by exp_manager
  logger: false # Provided by exp_manager
  log_every_n_steps: 100
  check_val_every_n_epoch: 5
  benchmark: false
  max_epochs: 100    #modified,default N/A
```

Although the default hifigan configuration also specifies a min_lr, it still reaches the 2e-4 rate set by adamw , which my fspitch setting couldn't. Could this be due to the difference between the cosine and noam decay strategies?

Moreover, I am aware that the maximum steps for the default hifigan exceed what my dataset can reach. Nevertheless, the results obtained without learning rate decay were not disappointing. I would like to know if the default setting expects annealing to be performed?