eval_trainer.predict() gives AttributeError: 'PipelineMixedPrecisionPlugin' object has no attribute '_desired_input_dtype' error
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

I encountered the following error when calling trainer_eval.predict() to test the fine-tuned LLM Model:

```
AttributeError                            Traceback (most recent call last)
Cell In[33], line 1
----> 1 response = trainer_eval.predict(model_eval, request_dl)
      2 for batch in response:
      3     for s in batch['sentences']:

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:864, in Trainer.predict(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
    862 self.state.status = TrainerStatus.RUNNING
    863 self.predicting = True
--> 864 return call._call_and_handle_interrupt(
    865     self, self._predict_impl, model, dataloaders, datamodule, return_predictions, ckpt_path
    866 )

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py:44, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     42     if trainer.strategy.launcher is not None:
     43         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
---> 44     return trainer_fn(*args, **kwargs)
     46 except _TunerExitException:
     47     _call_teardown_hook(trainer)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:903, in Trainer._predict_impl(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
    899 assert self.state.fn is not None
    900 ckpt_path = self._checkpoint_connector._select_ckpt_path(
    901     self.state.fn, ckpt_path, model_provided=model_provided, model_connected=self.lightning_module is not None
    902 )
--> 903 results = self._run(model, ckpt_path=ckpt_path)
    905 assert self.state.stopped
    906 self.predicting = False

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:987, in Trainer._run(self, model, ckpt_path)
    982 self._signal_connector.register_signal_handlers()
    984 # ----------------------------
    985 # RUN THE TRAINER
    986 # ----------------------------
--> 987 results = self._run_stage()
    989 # ----------------------------
    990 # POST-Training CLEAN UP
    991 # ----------------------------
    992 log.debug(f"{self.__class__.__name__}: trainer tearing down")

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:1028, in Trainer._run_stage(self)
   1026     return self._evaluation_loop.run()
   1027 if self.predicting:
-> 1028     return self.predict_loop.run()
   1029 if self.training:
   1030     with isolate_rng():

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py:182, in _no_grad_context.<locals>._decorator(self, *args, **kwargs)
    180     context_manager = torch.no_grad
    181 with context_manager():
--> 182     return loop_run(self, *args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/prediction_loop.py:124, in _PredictionLoop.run(self)
    122     self.batch_progress.is_last_batch = data_fetcher.done
    123     # run step hooks
--> 124     self._predict_step(batch, batch_idx, dataloader_idx, dataloader_iter)
    125 except StopIteration:
    126     # this needs to wrap the `*_step` call too (not just `next`) for `dataloader_iter` support
    127     break

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/prediction_loop.py:229, in _PredictionLoop._predict_step(self, batch, batch_idx, dataloader_idx, dataloader_iter)
    226 assert data_fetcher is not None
    228 if not (using_dataloader_iter := isinstance(data_fetcher, _DataLoaderIterDataFetcher)):
--> 229     batch = trainer.precision_plugin.convert_input(batch)
    230     batch = trainer.lightning_module._on_before_batch_transfer(batch, dataloader_idx=dataloader_idx)
    231     batch = call._call_strategy_hook(trainer, "batch_to_device", batch, dataloader_idx=dataloader_idx)

File /usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/fsdp.py:131, in FSDPPrecision.convert_input(self, data)
    129 @override
    130 def convert_input(self, data: Any) -> Any:
--> 131     return apply_to_collection(data, function=_convert_fp_tensor, dtype=Tensor, dst_type=self._desired_input_dtype)

AttributeError: 'PipelineMixedPrecisionPlugin' object has no attribute '_desired_input_dtype'

```
**Steps/Code to reproduce bug**

I'm following the steps and guides outlined in the [LoRA.ipynb](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/lora.ipynb) notebook example of finetuning an LLM (megatron_gpt). However, I encountered the following error when testing/evaluating the finetuned model by executing the following cell:

```python
response = trainer_eval.predict(model_eval, request_dl)
for batch in response:
    for s in batch['sentences']:
        print(f"{s}\n\n")
```

**Expected behavior**

getting output prediction from the fine-tuned model.

I also have a question regarding how to merge the adapter with the base model and save them in a single .nemo file for deployment. Could you please assist?

**Environment overview (please complete the following information)**

pulling the nemofw-training:24.03 image container
on A100 GPU


