Multiple last checkpoints
This happens when the job is killed while saving a checkpoint. Currently you need to manually cleanup the checkpoints folder to be able to resume. Figure out which is the most recent *complete* "-last" checkpoint, and delete the other(s). pls re-open if solution doesn;t work or if you face any other issue.  @szhengac To allow PTL to exit gracefully, if your job has some max runtime, you could tell PTL to exit earlier, so that the program has buffer time to clean up the old checkpoint. For example, if your job can only run for max 4 hours, you could set `++trainer.max_time="00:03:50:00"`. Instead of trainer.max_time, you should use exp_manager.max_time_per_run  

https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L175
Received the following error when I killed the job and try to resume it. It uses mcore gpt.
```
Traceback (most recent call last):
  File "/opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py", line 34, in main
    exp_manager(trainer, cfg.exp_manager)
  File "/opt/NeMo/nemo/utils/exp_manager.py", line 348, in exp_manager
    check_resume(
  File "/opt/NeMo/nemo/utils/exp_manager.py", line 622, in check_resume
    raise ValueError(f"Multiple checkpoints {last_checkpoints} that matches *last.ckpt.")
ValueError: Multiple checkpoints [PosixPath('/results/checkpoints/boson-consumed_samples=6224.0-last'), PosixPath('/results/checkpoints/boson-consumed_samples=4800.0-last')] that matches *last.ckpt.
```