Fail to load SpeakerNet when inference Speaker Diarization
**Describe the bug**

I trained SpeakerNet with my own dataset and got checkpoint to test speaker diarization. I tried both configs of SpeakerNet in here.
[Configs](https://github.com/NVIDIA/NeMo/tree/main/examples/speaker_recognition/conf)
Because pytorch lightning trainer save checkpoint as .ckpt format so i using model.load_from_checkpoint to load trained model.


**Code to reproduce bug**

```
def _init_speaker_model(self):
    model_path = self._cfg.diarizer.speaker_embeddings.model_path
    if model_path is not None and model_path.endswith('.nemo'):
        self._speaker_model = ExtractSpeakerEmbeddingsModel.restore_from(model_path)
        logging.info("Speaker Model restored locally from {}".format(model_path))
    elif model_path.endswith('.ckpt'):
        self._speaker_model = EncDecClassificationModel.load_from_checkpoint(model_path)
        logging.info("Speaker model loaded locally from {}".format(model_path))
    else:
        logging.info("Speaker model not available")
```

This is error log:
```
  File "infer.py", line 42, in <module>
    sd_model = ClusteringDiarizer(cfg=config)
  File "/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 92, in __init__
    self._init_speaker_model()
  File "/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 107, in _init_speaker_model
    self._speaker_model = EncDecClassificationModel.load_from_checkpoint(model_path)
  File "/opt/conda/lib/python3.6/site-packages/nemo/core/classes/modelPT.py", line 445, in load_from_checkpoint
    **kwargs,
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/saving.py", line 158, in load_from_checkpoint
    model = cls._load_model_state(checkpoint, strict=strict, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/saving.py", line 198, in _load_model_state
    model = cls(**_cls_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/classification_models.py", line 53, in __init__
    self._update_decoder_config(self._cfg.decoder)
  File "/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/classification_models.py", line 489, in _update_decoder_config
    labels = self.cfg.labels
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 305, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 303, in __getattr__
    return self._get_impl(key=key, default_value=DEFAULT_VALUE_MARKER)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 361, in _get_impl
    node = self._get_node(key=key)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 383, in _get_node
    self._validate_get(key)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 136, in _validate_get
    key=key, value=value, cause=ConfigAttributeError(msg)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
    _raise(ex, cause)
  File "/opt/conda/lib/python3.6/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: Key 'labels' is not in struct
        full_key: labels
        reference_type=Optional[Dict[Union[str, Enum], Any]]
        object_type=dict
```
Any idea about this. Thanks a lot! 
