Interaction between sampler len and trainer on restarts.
This has been resolved, can be closed.
https://github.com/NVIDIA/NeMo/blob/adcc72bfbb0344a9c2cbcd1ece9897b1c846f6dd/nemo/lightning/data.py#L192-L203

When you restart a job, you typically set consumed_samples to a non-zero value. This has the side effect of changing the length returned of the sampler. If the trainer current step is restored to some value N < total, and the trainer uses `len(loader)` to re-compute remaining steps, then when you restart a job the current step is properly restored, but the length of your progress bar (total) shrinks by N so it's then `total - N`. Can you verify that on restart this does not happen? Ideally adding a stop/go test where a trainer job is artificially stopped at some step less than total and resumed, then the total number of steps and all relevant parameters are checked to ensure that resumption worked properly.

This was definitely an issue for us in BioNeMo1 that we "fixed" by changing the length calculation to not be dependent on `consumed_samples`, since it seemed that at the time at least the lightning Trainer used `len(dataloader)` at init time (even when restarting) to determine the length of the progress bar.