Issue running PEFT finetuning on a GPT2 model with TP=2 and SP =True
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**
i am trying to run the GPT2 PEFT finetuning on GPT2 models. I try to convert my TP=1 .nemo file to TP=2 .nemo file using megatron_change_num_partitions. However, when I try to load the checkpoints from the .nemo file i get the following error.

A clear and concise description of what the bug is.

`Traceback (most recent call last):
  File "/scratch/avi2011/nemo_proj/test.py", line 137, in <module>
    model = MegatronGPTSFTModel.restore_from(config.model.restore_from_path, model_cfg, trainer=trainer)
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/models/nlp_model.py", line 465, in restore_from
    return super().restore_from(
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/classes/modelPT.py", line 442, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/parts/nlp_overrides.py", line 700, in restore_from
    loaded_params = super().load_config_and_state_dict(
  File "/usr/local/lib/python3.10/dist-packages/nemo/core/connectors/save_restore_connector.py", line 169, in load_config_and_state_dict
    state_dict = self._load_state_dict_from_disk(model_weights, map_location=map_location)
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/parts/nlp_overrides.py", line 663, in _load_state_dict_from_disk
    raise ValueError(f'Expected {model_weights} to be a file or directory.')
ValueError: Expected /state/partition1/job-44012513/tmpl8ypyvcd/model_weights.ckpt to be a file or directory.`


