Can't load MSDD model trained together with speaker embs extractor "speaker_model_cfg is not in struct"
Solved the issue, by providing the ckpt path, could I ask why it doesn't work using ".nemo" files? This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. I've also met this problem and solved by providing `.ckpt`, not the `.nemo`.  
I wonder what is the difference between both files.
Hi, I have trained an msdd model together with it's speaker embeddings extractor unfrozen. When I try to run:

```
python3 neural_diarizer/multiscale_diar_decoder_infer.py diarizer.manifest_filepath=manifests/dev.json diarizer.oracle_vad=True diarizer.out_dir=test diarizer.msdd_model.model_path=train_msdd/msdd_experiments/exp1/checkpoints/exp1.nemo
```

I get the following error:
```
[NeMo E 2024-02-20 13:28:37 common:505] Model instantiation failed!
    Target class:       nemo.collections.asr.models.msdd_models.EncDecDiarLabelModel
    Error(s):   Key 'speaker_model_cfg' is not in struct
        full_key: speaker_model_cfg
        object_type=dict
    Traceback (most recent call last):
      File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/classes/common.py", line 484, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/collections/asr/models/msdd_models.py", line 136, in __init__
        self.msdd._speaker_model = EncDecSpeakerLabelModel.from_config_dict(cfg.speaker_model_cfg)
      File "/usr/local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
        self._format_and_raise(key=key, value=None, cause=e)
      File "/usr/local/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
        format_and_raise(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
        _raise(ex, cause)
      File "/usr/local/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
        raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
      File "/usr/local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
        return self._get_impl(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
        node = self._get_child(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
        child = self._get_node(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
        self._validate_get(key)
      File "/usr/local/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
        self._format_and_raise(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
        format_and_raise(
      File "/usr/local/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
        _raise(ex, cause)
      File "/usr/local/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
        raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    omegaconf.errors.ConfigAttributeError: Key 'speaker_model_cfg' is not in struct
        full_key: speaker_model_cfg
        object_type=dict

Error executing job with overrides: ['diarizer.manifest_filepath=manifests/dev_displace24.json', 'diarizer.oracle_vad=True', 'diarizer.out_dir=test', 'diarizer.msdd_model.model_path=train_msdd/msdd_experiments/exp1/checkpoints/exp1.nemo']
Traceback (most recent call last):
  File "/home/gpirlogeanu/git_repos/diarization_nemo/diarization/neural_diarizer/multiscale_diar_decoder_infer.py", line 32, in main
    diarizer_model = NeuralDiarizer(cfg=cfg)
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/collections/asr/models/msdd_models.py", line 982, in __init__
    self._init_msdd_model(cfg)
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/collections/asr/models/msdd_models.py", line 1074, in _init_msdd_model
    self.msdd_model = EncDecDiarLabelModel.restore_from(restore_path=model_path)
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/classes/modelPT.py", line 316, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/connectors/save_restore_connector.py", line 235, in restore_from
    loaded_params = self.load_config_and_state_dict(
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/connectors/save_restore_connector.py", line 158, in load_config_and_state_dict
    instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/classes/common.py", line 506, in from_config_dict
    raise e
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/core/classes/common.py", line 498, in from_config_dict
    instance = cls(cfg=config, trainer=trainer)
  File "/home/gpirlogeanu/git_repos/diarization_nemo/nemo/collections/asr/models/msdd_models.py", line 136, in __init__
    self.msdd._speaker_model = EncDecSpeakerLabelModel.from_config_dict(cfg.speaker_model_cfg)
omegaconf.errors.ConfigAttributeError: Key 'speaker_model_cfg' is not in struct
    full_key: speaker_model_cfg
    object_type=dict
```
When I run inside the EncDecDiarLabel class:
`print(msdd_model.msdd._speaker_model)`
It returns None.

Could you please help?