Unfinished checkpoints handling
jenkins Hi @titu1994,

Thank you for the review! 
I added the docstrings and comments.

Regarding your question: If all checkpoints gets filtered out in Exp Manager, then depending on `resume_ignore_no_checkpoint` option  - training starts from scratch (resume_ignore_no_checkpoint=True) or raises NotFoundError (resume_ignore_no_checkpoint=False). @jbieniusiewi thanks for the PR. PTL 2.1 creates a symlink for the last checkpoint so we wouldn't have an issue where saving last checkpoint is incomplete. Would this PR be still relevant with 2.1 ?
CC @titu1994 @ericharper  Hi @athitten
I think some parts of this PR might be still relevant even with symlinks to the latest checkpoint introduced in TPL 2.1 E.g., incomplete checkpoints cleanup. Certainly this PR will need to be reviewed for PTL 2.1. 

  This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This PR was closed because it has been inactive for 7 days since being marked as stale. jenkins jenkins @titu1994 is this good to be merged or do you have any comments to be resolved ? This will need to be rechecked for PTL 2.1 release when we update 
# What does this PR do ?

Handle incomplete checkpoints.

With this fix, if last checkpointing operation was interrupted for some reason and left incomplete data, training should be resumed from the last available full checkpoint, instead of trying to read the incomplete checkpoint and crashing. This should simplify training auto-resume logic.

Implementation overview:
Create "unfinished" marker file for a checkpoint that is being written or removed. 
The marker file is deleted after the checkpoint IO operation completes.
When resuming, checkpoints marked as unfinished are ignored and subsequently removed.

**Collection**:
Core

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

@ericharper, @titu1994, @blisc, @okuchaiev