GPU memory overhead when calling trainer.fit the second time
**Describe the bug**

when calling `trainer.fit` the second time to train a NeMo model like:
```
trainer.fit(model)
trainer.fit_loop.epoch_loop.max_steps += 10
trainer.fit(model)
```
The second time would use more GPU memory than the first.

**Steps/Code to reproduce bug**
Create a `MegatronGPTModel` with `megatron_amp_O2=True` or with `distributed_fused_adam` and then use the above code for training.

When we print `torch.cuda.memory_allocated()` at the training step, we will find the GPU memory usage at the second `trainer.fit` is significantly higher.

**Expected behavior**

We expect the second `trainer.fit` should have the similar GPU memory usage to that in the first `trainer.fit`.

**Environment details**

Tried official Nemo v1.20 docker and it has this issue.
