assertion error on calling model_eval.load_adapters() in lora tutorial
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. Were you able to fix this? Thank you for checking in. unfortunately, I haven't been able to fix it.
**Describe the bug**
assertion error on calling model_eval.load_adapters() in lora tutorial


**Steps/Code to reproduce bug**

I'm basically just running every cells in [lora tutorial](https://github.com/NVIDIA/NeMo/blob/11b7a733cbd4b8311eacba581323f88c7cd4bac4/tutorials/nlp/lora.ipynb)
and got this error on the inference part.

the error seems to be from this line of code 
`model_eval.load_adapters(config_eval.model.peft.restore_from_path, LoraPEFTConfig(eval_model_cfg))`

error
```
AssertionError                            Traceback (most recent call last)
Cell In[32], [line 2](vscode-notebook-cell:?execution_count=32&line=2)
      [1](vscode-notebook-cell:?execution_count=32&line=1) model_eval = MegatronGPTSFTModel.restore_from(config_eval.model.restore_from_path, eval_model_cfg, trainer=trainer_eval)
----> [2](vscode-notebook-cell:?execution_count=32&line=2) model_eval.load_adapters(config_eval.model.peft.restore_from_path, LoraPEFTConfig(eval_model_cfg))
      [3](vscode-notebook-cell:?execution_count=32&line=3) model_eval.freeze()
      [5](vscode-notebook-cell:?execution_count=32&line=5) print("Parameter count manually:\n", model_eval.summarize())

File /opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py:296, in NLPAdapterModelMixin.load_adapters(self, filepath, peft_cfgs, map_location)
    [294](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f4e654d6f32227d.vscode-resource.vscode-cdn.net/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py:294)     peft_cfgs = [PEFT_CONFIG_MAP[conf.peft.peft_scheme](conf)]
    [295](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f4e654d6f32227d.vscode-resource.vscode-cdn.net/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py:295) self.add_adapter(peft_cfgs)
--> [296](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f4e654d6f32227d.vscode-resource.vscode-cdn.net/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py:296) assert set(state_dict.keys()) == self.adapter_keys.union(self.tunable_base_param_keys)
    [297](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f4e654d6f32227d.vscode-resource.vscode-cdn.net/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py:297) super().load_state_dict(state_dict, strict=False)

AssertionError:
```



**Environment overview (please complete the following information)**

 - Environment location: NeMo Docker container on WSL2 Ubuntu
 - Method of NeMo install: docker pull nvcr.io/nvidia/nemo:24.01.01.framework && docker run -it -d --name my_nemo_container --gpus all -p 8888:8888 -p 6006:6006 nvcr.io/nvidia/nemo:24.01.01.framework
**Environment details**

NVIDIA docker image

**Additional context**

GPU Model: RTX4070 Super
