speaker diarization inference Error: Missing 'speaker_model_cfg' Key in Configuration When Using Pretrained CKPT File
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
When running inference with the `multiscale_diar_decoder_infer.py` script after training the model, an error occurs related to a missing configuration attribute (`speaker_model_cfg`). The error traceback indicates that the configuration used for inference does not contain the required `speaker_model_cfg` key.

**Steps/Code to reproduce bug**  
1. Train the model using the following command:
   ```bash
   ./multiscale_diar_decoder.py --config-path=../conf/neural_diarizer --config-name=msdd_5scl_15_05_50Povl_256x3x32x2.yaml model.train_ds.manifest_filepath=/data2/xxx_data/SD/nemo/msdwild/train/msdd_data.50step.json model.validation_ds.manifest_filepath=/data2/xxx_data/SD/nemo/msdwild/test/msdd_data.50step.json model.train_ds.emb_dir=/data2/xxx_data/SD/nemo/msdwild/train model.validation_ds.emb_dir=/data2/xxx_data/SD/nemo/msdwild/test exp_manager.name=sample_train exp_manager.exp_dir=./msdd_exp
   ```
2. After training, use the following command to run inference:
   ```bash
   python multiscale_diar_decoder_infer.py --config-path='../conf/inference' --config-name='diar_infer_telephonic.yaml' \
   diarizer.msdd_model.model_path='/home/data/xxx/project/SD/NeMo/examples/speaker_tasks/diarization/neural_diarizer/msdd_exp/sample_train/2024-12-17_12-32-23/checkpoints/test.ckpt' \
   diarizer.oracle_vad=True \
   diarizer.manifest_filepath='/data2/xxx_data/SD/nemo/msdwild/train/msdd_data.50step.json' \
   diarizer.out_dir='output'
   ```
3. The following error occurs during inference:
   ```
   Traceback (most recent call last):
   File "/home/data/xxx/project/SD/NeMo/examples/speaker_tasks/diarization/neural_diarizer/multiscale_diar_decoder_infer.py", line 31, in main
   diarizer_model = NeuralDiarizer(cfg=cfg).to(cfg.device)
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/models/msdd_models.py", line 1019, in __init__
   self._init_msdd_model(cfg)
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/models/msdd_models.py", line 1114, in _init_msdd_model
   self.msdd_model = EncDecDiarLabelModel.load_from_checkpoint(
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/nemo/core/classes/modelPT.py", line 506, in load_from_checkpoint
   checkpoint = super().load_from_checkpoint(
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/lightning/pytorch/utilities/model_helpers.py", line 125, in wrapper
   return self.method(cls, *args, **kwargs)
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 1582, in load_from_checkpoint
   loaded = _load_from_checkpoint(
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/lightning/pytorch/core/saving.py", line 91, in _load_from_checkpoint
   model = _load_state(cls, checkpoint, strict=strict, **kwargs)
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/lightning/pytorch/core/saving.py", line 165, in _load_state
   obj = instantiator(cls, _cls_kwargs) if instantiator else cls(**_cls_kwargs)
   File "/home/data/xxx/anaconda3/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/models/msdd_models.py", line 145, in __init__
   self.msdd._speaker_model = EncDexxxeakerLabelModel.from_config_dict(cfg.speaker_model_cfg)
   omegaconf.errors.ConfigAttributeError: Key 'speaker_model_cfg' is not in struct
   full_key: cfg.speaker_model_cfg
   object_type=dict
   ```

I found a similar issue in #8463, but I am passing the ckpt file and the issue is not resolved.
Could you please help?