speech_to_text_finetune `hparams` not updated
Hi, 
I noticed the same thing, here a few more things I noted:
- 'log_prediction' and 'decoding_strategy' are not used if specify
- Hparams file of the experiment directory seems to be the values from the model config file
- Learning rate seems to work because initial_lr of the model is what I specify in my config (because it is updated afterward with "asr_model.setup_optimization(cfg.model.optim)")
- specifying 'preprocessor' in the config file does nothing it seems because when changing 'pad_to' parameter the logger still log "PADDING: 0" (where 0 is the value from the model config file) instead of the value I set. 

When loading the model it doesn't look at the config we do, it only looks at the model config file, so that might be the problem. It is then overridden by various functions like "setup_optimization". As a workaround, I copied the model config file (to be sure to have all parameters), then I change what I want, and finally I pass this new file as override of the model config in the line below

```asr_model = ASRModel.from_pretrained(model_name=pretrained_name, trainer=trainer, override_config_path=override_config_path)```
 Hi, sorry for the confusion. The `examples/asr/speech_to_text_finetune.py` script is only intended for changing the dataset, optim, spec_augment, vocab/tokenizer of the model. Other changes related to the model architecture should go to the main ASR training scripts `speech_to_text_rnnt/ctc_*.py` and use `init_from_nemo_model ` or `init_from_pretrained_model` to load a pretrained checkpoint. @stevehuang52 Yes, I am aware. And I am using the script with that intention exactly. I have used `speech_to_text_rnnt/ctc_*.py` before but since I am not changing the architecture, the configs were just bloated so I am trying to migrate to `speech_to_text_finetune.py`.

Unfortunately the script does not update the hprams and/or related correctly. This is a big deal for experiment tracking. I have the same problem. I am using hf dataset for finetune. Optim parameters are not updating. Is there any update on this?  This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. @stevehuang52 @nithinraok Can we please get an update? sorry for the delay, the bug will be fixed in https://github.com/NVIDIA/NeMo/pull/10508

CC @nithinraok @titu1994  Nice. Thanks everyone! This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

When using the `examples/asr/speech_to_text_finetune.py` to finetune an ASR model with a finetuning config modified from the example `examples/asr/conf/asr_finetune/speech_to_text_finetune.yaml`.

```yaml
model:
  sample_rate: 16000
  log_prediction: true
  rnnt_reduction: 'mean_volume'
  ctc_reduction: 'mean_volume'
  skip_nan_grad: false

 ...

  optim:
    name: adamw
    lr: 1e-5
    # optimizer arguments
    betas: [ 0.9, 0.98 ]
    weight_decay: 1e-3
```

`ctc_reduction` has no effect.
`asr_model.hparams["cfg"]["ctc_reduction"]` is `mean_batch` like in the original model.

`optim` seems to have no effect
`asr_model.hparams["cfg"]["optim"]` is `{'betas': [0.9, 0.98], 'lr': 3.0, 'name': 'adamw', 'sched': {'d_model': 1024, 'min_lr': 1e-06, 'name': 'NoamAnnealing', 'warmup_ratio': None, 'warmup_steps': 5000}, 'weight_decay': 0.001}`

`asr_model.hparams["cfg"]["nemo_version"]` is not updated (set to 1.20.0, finetuning done with 1.23.0)

Evaluated right before `trainer.fit(asr_model)`.

When logged in wandb the config is as above BUT the logged learning rate actually follows the target config, not the original config.

**Steps/Code to reproduce bug**

1. Download pretrained model https://catalog.ngc.nvidia.com/orgs/nvidia/teams/riva/models/parakeet-ctc-riva-0-6b-unified-en-us/files
2. Write a basic manifest file as above
3. `python speech_to_text_finetune.py --config-path=./configs/ --config-name=[name]`
4. If using wandb, the config will be logged with the bugs explained above
5. If not using wandb, breakpoint before train and run `asr_model.hparams`

**Expected behavior**

`asr_model.hparams["cfg"]["optim"]` should match the finetuning config
`asr_model.hparams["cfg"]["nemo_version"]` should be updated to the nemo version used for finetuning

**Environment overview (please complete the following information)**

 - Environment location: Docker on AWS `g5.2xlarge`
 - Method of NeMo install: pip intall

Dockerfile

```Dockerfile
ARG BASE_IMAGE=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker
FROM $BASE_IMAGE AS base

# Set the working directory to /opt/ml/code for Sagemaker
WORKDIR /opt/ml/code

# Assumes we run the docker build from ai/asr/nemo folder
COPY ./code .

FROM base AS cert

FROM cert AS requirements

WORKDIR /opt/ml/code
RUN pip install Cython  # needed for NeMo
RUN pip install -r ./requirements.txt

FROM requirements AS final

WORKDIR /opt/ml/code
CMD bash
```

requirements.txt content

```text
huggingface-hub==0.23.5
nemo_toolkit[all]==1.23.0
numpy==1.23.5
torchmetrics==0.11.4
```

**Environment details**

torch==2.0.1
python==3.10.8

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:          20.04
Codename:     focal


**Additional context**

CUDA Driver Version: 12.3
GPU: A10g
