training resume is not working in 24.07 container
Thank you for the bug report. We're still working on a patch. Hi,

this issue is fixed in our dev nightly container:
[nvcr.io/nvidian/nemo:nightly](http://nvcr.io/nvidian/nemo:nightly)
**Describe the bug**

In version 24.05, the training continuation process was working smoothly. However, after upgrading to version 24.07, when resuming after a pause, the loss value spikes significantly.

This seems to be a highly critical issue that needs investigation. The problem occurs consistently, regardless of the model size or data, and resuming training is failing 100% of the time.

**Steps/Code to reproduce bug**

1. Perform llama 7b **pretraining** with a mock dataset
2. Save some checkpoints
3. Resume from the saved checkpoint
4. Loss spikes

**Expected behavior**

working normally

**Environment overview (please complete the following information)**

- using 24.07 docker container
- H100