How to make checkpoints self-contained per step instead of per rank?
In megatron-lm checkpoints are self-contained, so if I have a checkpoint for step 1000, all its files are in the same directory. However in nemo they are grouped by rank, rather than step.

This causes a huge problem when one needs to sync checkpoints to cloud and be able to delete them once this is done.

Is there a way to invert the checkpoint structure, so that instead of:

```
checkpoints/mp_rank_00/step=02250.ckpt
checkpoints/mp_rank_00/step=02500.ckpt

checkpoints/mp_rank_01/step=02250.ckpt
checkpoints/mp_rank_01/step=02500.ckpt
```
it'll save checkpoints like:
```
checkpoints/02250/mp_rank_00.ckpt
checkpoints/02250/mp_rank_01.ckpt

checkpoints/02500/mp_rank_00.ckpt
checkpoints/02500/mp_rank_01.ckpt
```
So now I can copy+delete `checkpoints/02250` and automate this process easily by discovering new checkpoint folders, rather than trying to scan multiple leaf points, which makes it much more complicated and not lending to simple manual deleting.

I guess that since PTL's Experiment Manager is used this is an artifact of having each gpu saving into its unique rank path.

Any suggestions to how to solve it to bring back megatron-lm style checkpoints?

Thank you!