Resuming from a checkpoint that ended before the epoch ended and your dataloader is not resumable
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

I don't know if it is a real issue but I have the following warning while resuming a training where the checkpoint is not saved at the end of the epoch. I don't know what it means by "unreliable results" since the training seems to start at the batch it was when the checkpoint was made.

```
[NeMo W 2024-10-08 11:49:10 nemo_logging:349] /home/.../.local/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py:161: 

You're resuming from a checkpoint that ended before the epoch ended and your dataloader is not resumable. This can cause unreliable results if further training is done. Consider using an end-of-epoch checkpoint or make your dataloader resumable by implementing the `state_dict` / `load_state_dict` interface.
```

**Steps/Code to reproduce bug**

I think you can use [this](https://github.com/NVIDIA/NeMo/blob/ff97c70fc9d7a0050c947b9008372bcf2e6f3b48/examples/asr/asr_ctc/speech_to_text_ctc.py) to try since it is what I'm based on.
I used a tarred dataset (it does it with a non tarred dataset too) with the following options in the exp_manager:
```yaml
exp_manager:
  exp_dir: null
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  create_early_stopping_callback: False
  early_stopping_callback_params:
    check_finite: False
    patience: 5
    mode: "min"
    monitor: "val_loss"
  checkpoint_callback_params:
    every_n_epochs: null
    every_n_train_steps: 8
    monitor: train_loss
    save_top_k: -1
    always_save_nemo: false
    save_last: true
    filename: testing_filename-{epoch:02d}-{step:06d}-{train_loss:.4f}
  resume_if_exists: true
  resume_ignore_no_checkpoint: true
```


**Expected behavior**

Should not throw a warning

**Environment details**

I locally installed NeMo with pip and I have a Nvidia GPU
