GPU memory overhead when calling trainer.fit the second time
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

Reopen the issue https://github.com/NVIDIA/NeMo/issues/7833 

**Steps/Code to reproduce bug**

when calling `trainer.fit` the second time to train a NeMo model like:
```
trainer.fit(model)
trainer.fit_loop.epoch_loop.max_steps += 10
trainer.fit(model)
```
The second time would use more GPU memory than the first.

**Steps/Code to reproduce bug**
Create a `MegatronGPTModel` with `megatron_amp_O2=True` or with `distributed_fused_adam` and then use the above code for training.

When we print `torch.cuda.memory_allocated()` at the training step, we will find the GPU memory usage at the second `trainer.fit` is significantly higher.

**Expected behavior**

We expect the second `trainer.fit` should have the similar GPU memory usage to that in the first `trainer.fit`.

**Environment details**

Check the official Nemo v1.20 docker and NeMo v1.22 branch, both have this issue.
